These last three years have been years like no other, yet we hope this finds you and your family safe from the virus that has uprooted our daily lives. Who really knows what the New Year has in store for us all, but just know you will be able to count on us yet again. As 2022 comes to a close, we wish to thank you for allowing iTech Solutions (now part of Service Express) to be part of your team. We hope that our IBM i newsletters, Blogs, Webinars, Sips & Tricks, iTip Videos, Two-Day iAdmin Conferences, iBasics seminars, and iPower Hour Podcast have been educational and informative for you and that you have learned from them. We encourage your feedback on what other things we can add to make them more helpful to our customers, and others who read them. Look for even more to come this year in 2023.
With our acquisition by Service Express in July, we have continued our growth this year adding additional employees to improve our services as well as the breadth of our IBM i offerings. Having the backing of Service Express we have also been working on expanding our offerings, which is great news for the IBM i community. We hope you have a happy holiday season, with good health, happiness, and prosperity in the New Year. All of us at iTech thank you for your business, and for the confidence that you have placed in us over the years. We look forward to working together in the coming year, and we will continue to strive to exceed your expectations. Whenever we don’t, please reach out directly to me. While my job has changed, my ability to get you help will never change.
Remember the Christmas jingle:
The weather outside is frightful,
but working together with iTech Solutions can be so delightful.
Let it be IBM i, IBM i, IBM i.
Let us help you with all your IBM i needs next year. Now, sit back and chuckle as you read about how iTech Solutions helped Santa this Christmas :). We will be back in January with our normal IBM i System Administrator Technical Tips and Techniques.
All the best to you, your family, and your organization.
From all of us at iTech Solutions.
With just a few more days to shop, work, and wait for Christmas. I thought I would share a little story about how iTech Solutions helped Santa prepare for Christmas this year.
As many of our readers remember, Santa Claus and his operations at the North Pole are one of our prized customer accounts. We have been helping St. Nick administer his Power Systems since he was running on an AS/400 and our business partner relationship has encompassed many projects and all aspects of his IT infrastructure.
When we started supporting SLED (Santa Logistics/Enterprises Department), we were addressing issues as they impact the business. Now that we have been working together for years, we anticipate needs and address them before Christmas needs saving. Stress levels are far lower at the North Pole, and in appreciation, Santa sent me a bottle of Pappy Van Winkle Bourbon for my “hot chocolate” this year, I told him I prefer it just on the rocks.
Managed Service Provider
Last year, Santa decided to become an MSP customer. This refers to Managed Service Provider – we handle all of the facets of his IBM i on Power needs. PTFs, OS upgrades, backup, monitoring, recovery, replication. It is all the IBM i System Administration and infrastructure — basically everything except programming.
We have been meeting with him twice a month all year and we keep him updated on all our efforts. His elves come to Yvonne with requests for projects carried out by their primary and secondary technicians as they need support and we all function as a team together. All of us secured our places on the Nice List, even the salespeople! Jenks keeps an eye on things and responds as needed 7 days a week. Jim implemented MIMIX and now Santa has the peace of mind that comes from knowing his complete system is replicated to his DR site on the South Pole, as well as our highly secured Tier III Datacenter in our iInTheCloud environment for HA that Larry set up for him.
Nate was responsible to set up our system monitoring so there is nothing happening that we can’t resolve in real-time as we are notified immediately. Mike updated the firmware on his SAN and Tape drives. Mike was scanning his systems for Log4j, and reporting the results to the Head Elf. Victor was upgrading the firmware on his HMC and FSP to the latest level. Steven set up Full System FlashCopy on his new SAN so that we can do a flash copy of his entire partition every night. Then we bring the flashed partition up and do a full system save every night while the Elves still access his production system. Steve performed a security assessment and we have been diligently addressing all areas identified as needing remediation. Brett, Dan, and Mike keep an eye on all things during the second and third shifts, along with the weekends so that if something happens he is there in a flash to resolve it. Marc upgraded all his clients to Access Client Solutions, along with updating all his certifications. Just like the elves we work together and make magic happen! We even had an Elf on the Shelf in our Danbury Computer room, keeping an eye on some servers and insuring that all the System Administrators were on their best behavior.
VERIFi Automated Health Reporting
One thing Santa found very handy this year, was his Automated Weekly VERIFi report as well as VERIFi Security Advisor. He said it was like turning one of his elves into an IBM i System administrator who gave him a report each week on the status of his systems.
VERIFi is an iTech Solutions offering that will give you a weekly status report for all your IBM i partitions. Delivered like a single pane of glass, VERIFi covers processor and user license compliance, important dates like IBM End of Service for your machine, your partitions’ last SAVSYS and IPL, your PTF and OS currency, as well as a detailed statistics section of the last 7 days running to help identify potential problems before they start. It is a report that lets you know what is happening on your machine.
VERIFi Security Advisor is a proactive overview of your system’s security, with a semi-monthly security overview and forecasting. It also includes an annual one-hour advisory session with an iTech representative. VERIFi Security Advisors delivers reporting on things like system values, audit journal, invalid password attempts, encryption, user profile security, and more.
The best thing about VERIFi and VERIFi Security Advisor is that you don’t have to choose between one or the other. You can have both reports with VERIFi+. View all VERIFi options and pricing here.
Here is a sample report:
2022 sure has been a year for Santa’s naughty list! I think COVID-19 is at the top of the list. And it isn’t surprising that Santa is concerned about security so we made that our highest priority. Alicia and Steve had been doing a security assessment and review when Steve pointed out that he was sharing the root of his IFS directory so that he was exposed. In addition, we pointed out a few cyphers that his trading partners (you know all the big retail stores) were using that had been deprecated, and that they should be using newer more secure cyphers. What a year we had covering all the ways systems can be exploited! As you can imagine the servers at the North Pole are a big target, but you would be surprised how many ordinary businesses are under attack every day.
We started with user profiles and making sure the advanced authority was only in the hands of the elves that need it. Not long after, we found an internal threat from a disgruntled employee. Luckily his access had been restricted and we were monitoring for probes so we were able to determine the perpetrator. That led us to be concerned about the naughty and nice lists and we implemented Row and Column Access control to block out everyone except Santa and Mrs. Claus. Of course, the internal threat made us start thinking about monitoring for unauthorized modifications, so Donna set up a project to implement reviewing journal receivers and for added oversight, the Chief ELFormation Officer audits as well. This is when we also discovered the Log4j potential breach and gave Santa a game plan for locking things down.
We were quite surprised when someone added a new user called Hacker. At first, we thought it was one of the Reindeer. You know, Dancer, Prancer, Donner, Blixen, and the most famous reindeer of all Rodolph the Red-nosed reindeer. There was a new user profile in the Reindeer group profile called Hacker. The reindeer group has elevated privileges on all the North pole IBM I partitions, so we need to be careful. All of a sudden, through our security monitoring and exit point software we started getting alerts. It was first noticed overnight, and Mike had escalated his concern when he saw the alert and called Victor, who immediately shut down the user profile. It was a good thing, as Hacker was trying to upload a ransomware virus, which would have shut down Christmas. He was trying to use both a file share and FTP, but our software notified us both times. We were able to stop this before it caused any problems. These problems are happening all over the world, and as you will see below we are poised to help customers.
Then there is Wi-Fi access! That needed to be locked down so we hardened that pipeline like old-fashioned Christmas Candy. Any hacker trying to break in is going to break a tooth instead. iTech learned more than a thing or two from knowing when one is sleeping versus awake and we developed a new product in response to this inspiration. In that way, Santa has given our customers a gift in what we will offer them in 2023! IBM i Security Monitoring will allow us to keep your system safe, secure, and be immediately notified when a threat is detected.
iTech Solutions was at the North Pole in September doing a few OS upgrades. Santa is on our OS Subscription Service, which gives him 1 OS upgrade and 3 PTF applications over a 24-month period. He thought the price of only $295 a month per partition was quite reasonable. Finally, an old application that was running on 7.2 with Java 6 was updated. We had been keeping an eye on this closely as IBM i 7.2 went out of support on April 30, 2021, and if we hadn’t gotten it done his software maintenance on that machine would have doubled. We upgraded Santa’s POWER8 from 7.3 to 7.4 Technology Refresh 7 and one partition on his POWER9 that we upgraded from IBM i 7.4 to 7.5 TR 1. Santa Claus called iTech Solutions because he wanted someone with knowledge, experience, and know-how to do the operating system upgrades and he wanted the upgrades done right. He was asking me about when the next version of IBM i is going to be released, and I told him from the IBM i roadmap, it looks like some time in 2025.
A New Power10
Santa decided he deserved a Christmas present as well and a new Power10 9105-42A to replace the P8 was his choice. Last year, he kept telling me he wanted a Power10, and I told him only the enterprise-class machines were available. He kept arguing with me, telling me he heard all about the Power10 processor at the Hot Chips Conference and was sure that IBM had come out with Power10 servers already. I had to tell him they were just the large enterprise-class machines that were delivered, and that IBM would release the low-end machines in July. Now, with July in the rearview mirror, he was ready for his new Power10. Once we placed the order, we moved to implementation and migration. Donna set up a project plan for implementation, which was so detailed Santa thought Donna might have some Elf lineage in her. We had a meeting to discuss the project, and afterward, he knew in a snap that we had everything under control for a smooth installation and migration. We brought in a new Power10 S1024 16-Core machine to give Santa more performance as he had hit the limit on that old machine. We worked closely with his team to design exactly what they needed including NVMe (Non-Volatile Memory Express) which was 5 times faster than his old SSDs, more CPW per core, new PCIe adapter cards, extremely fast PCIe Gen4 bus, and more. Even better he received a new machine with 20x the performance of his existing machine for the same price he was paying for three years of maintenance! This reduced his 10-hour jobs at night to just minutes. Those NVMe devices fly. More time for him to spend with Mrs. Claus.
Santa said to me, “I can’t believe how smooth the migration went. Yesterday we were on the old Power8 machine, and today we are on the new Power10 machine. Everything is there and working, with the only difference being the speed. This has been the best migration ever, I am glad I chose iTech Solutions”. Whew, I escaped being placed on the naughty list.
Having the right project plan and business partner will surely make the project go smoothly.
iTech Solutions Upgrade Assurance
Of course, Project SLED at the North Pole can’t afford any downtime, so Santa uses the iTech Solutions Upgrade Assurance program, which is where we bring one of our machines up to the North Pole. We restored the XMAS-01 partition running 7.4 to the iTech machine and upgraded that to 7.5. Alicia setup a project plan with all the details of the test. This way all the applications were tested and once we ensured all was proper then we performed the work on Santa’s machine getting XMAS-01 to 7.5. Santa knows that there is no need to worry when he has iTech keeping his systems in tip-top shape.
After one of our monthly MSP meetings over the summer, Santa had the elves bring us some hot chocolate, I asked if he could put a double shot of bourbon in mine. He raised an eyebrow and asked, “Do you want to be put back on the naughty list?” I replied back, “Extra whip cream will be fine”. He smiled back, but then said he had another problem. He told me that he was always wishing he could go back in time, and run a query as if it was a few months earlier. I looked at him with a smile and said, “Do you wish you could be querying the files as if it was the middle of June, even though today is Dec 20th?”
This was exactly what he needed as the elves are constantly doing restores of the previous year’s gift and naughty/nice files to accomplish this task. Now, as we had implemented BRMS for Santa a few years ago doing the restores wasn’t too difficult, however, 7.3 makes it even easier to query the data! I explained Temporal Support, which at first he thought was the yummy fried coating we had on those shrimp at the Japanese restaurant. “No, no, no, that is Tempura”, I said. “This is Temporal, which means time.” It’s like going back in history and seeing the data in the files, exactly how it existed on any particular day and time. Santa said that he could use this function to keep track of the many lists he has without having to constantly make copies of files to then be queried later. He sure has embraced the value of modernizing his IT!
It wouldn’t surprise anyone to know that it is a mighty big database residing in DB2. Keeping track of every child in the world is quite the task and without the reliability of IBM i Santa would be struggling. The Power infrastructure ensures that his database files interact seamlessly with a variety of applications. First, there is his ERP package for all the sourcing from vendors for materials to make the toys. Then there is the reverse packing distribution system that ensures all the gifts are put on the sleigh in the exact opposite order as they will be delivered. Another benefit of using 7.5 is running old queries using Query/400 & OPNQRYF under SQE vs CQE resulting in his queries running in almost 1/2 the time. I love it when the fat guy has that jolly old belly laugh.
WebSphere Application Server
Of course, he takes good care of all his employees, so his reindeer health application is important. This runs in WebSphere Application Server which Steve upgraded to WebSphere 188.8.131.52. His weather application lets him know where the sleigh might catch some air turbulence runs on JAVA so we updated that to Java 11. Most of the Elves love running green screen applications for the speed, we talked about enhancing some of those applications with a GUI interface using RPG Open Access. This is going to let him extend his RPG applications with accessibility for the new iPhone 14s & iPad Air Pro devices. Mobile applications are a must for the sleigh and Santa needs to keep in constant communication with the North Pole on Dec 24th. He was so happy when iTech used IBM i 7.5 to run Access Client Solutions to display directly to the mobile devices on the sleigh. I can’t remember who I heard mumbling it just needs Java, that’s it. Wow. In addition, Marc set him up with OpenJDK, so he didn’t have to pay the hostage fee to Oracle.
PHP, Ruby, Node.js, and Python
Another advantage of becoming an MSP customer is we meet monthly all year round, not just when there is a large-scale project needed. This allows us to take advantage of slower times to implement strategic initiatives. The younger elves are getting excited about website development with Ruby and using Python to take advantage of a community-based software development model to reduce overhead. However, other elves want Node.js for developing server-side web applications. Santa was concerned about the cost of managing more machines to host all these languages. I was able to tell him he could relax and not to worry, iTech Solutions has your back. We can install all those languages on your IBM i. At first, he thought I had been nipping on the eggnog as this seemed too good to be true. However, true to my promise to stay off the naughty list that wasn’t the case. IBM i now runs all these open source languages like PHP, Ruby, Node.js, Python, Java, and PHP natively. He was amazed. We setup MERLIN for his application development, and then it didn’t matter which application language the Elves were programming in.
Maintenance on his old machines
Santa has a few old Power6 and Power7 machines that he continues to run because the applications on them need IBM i 6.1 and 7.1. I believe it has something to do that these applications use older versions of WebSphere and Java, and the vendor went out of business so without the source it is difficult to migrate them to later releases of Java. So, we are running these old machines with these old versions of the operating system. Well, these are very important applications, and IBM wouldn’t provide hardware maintenance for them any longer. I said, no worries. Now, that we are part of Service Express, the leader in 3rd party maintenance, we can put these old machines under maintenance for a real savings. I then continued, you know those old Dell Servers & Cisco switches you have in the south side of the computer room, we can put those under a Service Express maintenance contract. I had Richie work up a quote which Santa signed, and Santa was amazed at just how inexpensive we could maintain all those servers. A few months later I was asking one of the elves how it was working out, and he said, “We had a drive go bad, and the Service Express technician arrived on site with a drive, replaced it and I didn’t even know about it, if I hadn’t run into him in the hall”. Well, I said that is what I would have expected.
Leveraging Investment in IBM i
When Santa first engaged us we started with some performance tuning on the machines at the North Pole. He had purchased a used machine from another business partner to save money, who unfortunately didn’t understand logical partitions, and had the processors dedicated to each partition. Santa told me in retrospect, that machine cost him more money in the long run and should have gone with our proposal in the first place. We fixed this machine to use shared processor pools. So, on Christmas Eve when the production partition XMAS-01 was previously running at 100%, yet no one was on any of the other partitions, so they were completely idle. It just didn’t seem like it was implemented correctly. Now using shared processors and uncapped partitions on his machine, and the XMAS-01 partition was now running at 350%, and the work was flying through. Santa, said he was able to save an upgrade and spend that money on additional toys for girls and boys throughout the world, all because in his words “iTech Solutions understands how to leverage my investment in IBM i to get the most out of the machine. No one else has the knowledge to get my machine running as iTech Solutions”. Thanks for the kind words, Santa. This year, we upgraded that machine to a Power10 9105-42A, and installed IBM Power Enterprise Pools 2.0 on this and the other Power10. Enterprise Pools 2.0 provides enhanced multisystem resource sharing and by-the-minute consumption of on-premises compute resources so Santa can deploy and manage a private cloud infrastructure. All installed processors and memory on servers in a Power Enterprise Pool 2.0 are activated and made available for immediate use when a pool is started. Processor and memory usage on each server are tracked by the minute and aggregated across the pool, now Santa gets the processors and memory during the week of Christmas, and especially on Christmas eve, and can save money the rest of the 51 weeks a year. Santa is saving money because iTech Solutions understands the IBM product portfolio, and Santa uses that money for more toys and clothes for good children all over the world.
I’m sure the elves still haven’t changed their code from a long time ago; even though iTech told Santa it’s time to modernize when we were there last year. Those Elves are slowing down one of the POWER9 S914 servers with all of those OPNQRYF’s on the toy list and nice list too. They never did want to change over and learn SQL. They are starting to after we gave them a demo of how easy it is to use SQL, and the speed of the queries. Next, with free-format RPG and the new MERLIN development environment, even the elves, maybe Mrs. Claus, are making minor changes and efficient rewrites on the legacy RPG code with ease. I think right after Christmas, it will be time to relook at his overall machine strategy. We offer our customers our review and assessment to validate the resources and requirements on their machines. I remember it was just three years ago, we reviewed a machine that had spinning disks and easily changed those to SSDs and that made such a difference. We reviewed that same machine this year and changed the SSDs to NVMe, and the machine’s performance improved yet again. If you need a review of your machines, contact one of our salespeople at iTech Solutions.
IBM i Performance
Santa was telling me that we still needed better performance on his Power9 S924 that we installed two years ago. He was saying that as the population of the world increases, the sizes of his files were just getting huge. Some of his jobs were running all night. We looked at the performance of the machine and discovered the amount of I/O on that machine had grown extremely high. So, we installed a new IBM Storwize FS7300 SAN with FlashCore Modules (FCM). It had the speed, virtualization, and capacity that he needed. We knew he needed that additional cache that can be purchased for the SAN, and Wow, did that do the trick. Those same jobs which were so I/O intensive ran very quickly with the new SAN and allowed the mapping software which determines the route that Santa is to take on Christmas Eve, stopping at every house to drop off the presents to run in just an hour compared to days. If your system seems to not be running how you want, why not have iTech Solutions come in and do a performance review?
Our next performance issue had to do with some ODBC jobs & Queries that the Elves had created out in the shop. The elves have created more queries than you can imagine. Every time they ran these ODBC jobs and Queries, the system would just grind to a halt. Mike ran the Index Advisor and determined that a couple of indexes (logical files) needed to be created over the toy file. With a mouse click, we created the required logical files for Santa. Now those jobs the elves were running are done in a snap. Santa said, “It was like a new machine, having iTech Solutions come in for the day and investigate the system”. We performed our standard Health Check. Perhaps this is something you might want to put on your list for Christmas or even next year?
During my visit, I was in the office next to the computer room at the North Pole, when out in the computer room there arose such a clatter,
I sprang from my desk to see what was the matter.
Away from my Windows10 laptop, I flew like a flash,
Tore up the ramp and threw open the sash.
And what to my wondering eyes did appear,
but a miniature sleigh filled with hundreds of tapes, and eight tiny reindeer;
With a little old driver, so lively and quick,
I knew in a moment it must be Saint Nick.
Disaster Recovery and Backup
I said, “Santa, what is going on?” Santa said, “We are all going to do our DR test, and we had to get all the tapes for our IBM i restoration”. I said, “That is a lot of tapes. Santa we spoke about this during the summer, why not just replicate to one of our IBM i cloud (Power Systems) machines at iTech Solutions, and then not only would you have a Hot Site available all the time, if you had a real disaster your DR machine would be immediately available and you wouldn’t be stumbling with all these tapes. Remember we were talking about this whole Cloud Computing concept with the Easter Bunny, and you thought he had been drinking a little too much carrot juice.”
Cloud Computing is not computers in the clouds, it’s a service that iTech Solutions offers that allows you to host your IBM i machines at our locations or even just use one of our machines and carve out a virtual logical partition. Santa said, “I had a lot on my mind at the time with COVID, as the elves had to do the systems management of my current machine. I couldn’t find any good system admins in the North Pole.” We agreed to look into hosting his replication with iTech Solutions in January. I said to Santa, “You know while we are at it, you should really look at either our Cloud-based backup solution, where you back your data directly up to our cloud, fully encrypted and compressed. Your backup is immediately available off-site, and you can rest assured that the data is fully encrypted. In addition, we can always put a Cybernetics virtual backup appliance (VTL) in the data center, and perform backups to the Virtual Tape Library so that you aren’t handing tapes.” Santa was interested in further looking at both solutions. Are you still playing with tapes? Think about that for the New Year.
Next, I asked Santa if there was anything else on his list? He said, “there were three more things. First, he said, Mrs. Claus gets so annoyed at him every month when he has to go into the office to do a Full System Backup. I looked at Santa, and said to him, “I have been asking you for years to get rid of that Twin-ax Console on the POWER6 and upgrade to Operations LAN Console, or even better let’s use one of the new POWER9 machines that have the HMC attached. You kept telling me that you liked Twin-ax. I said last quarter when we were doing the iTech Solutions quarterly PTF maintenance to bring all your PTFs up to date, that I could hardly see that screen, as that twin-ax device is so old on that one Power6 machine. If you had an Operations LAN console, you, iTech Solutions, or any of the elves could remotely connect from home via VPN right to the console, and do the backup without having to come into the office”. iTech Solutions installed and configured Operations LAN Console so that Santa could spend a little more quality time with Mrs. Claus. Mrs. Claus was so happy she told Santa to put everyone at iTech Solutions on the Nice list, no matter what. Whew, that was certainly lucky for me.
Secondly, he said, “From time to time the elves are developing new modules and programs and it would be nice if he could get them off the production XMAS01 machine”. I said, “Well that isn’t a problem. Let’s create a virtual partition on that new POWER10 S1014 machine, and put the development environment on that partition”. “How could we do that?”, he asked. I said, “Remember when we brought that new POWER10 machine in? Now that we are running IBM i 7.5 we can do more virtualization and create virtual partitions.” Santa said to me, “But I only have one pair of disk controllers on XMAS01”. I said, “No problem. Remember when Steven setup VIOS to the new SAN? We are going to virtualize the LUNS with NPIV, and then share the SAS tape drive between partitions. Since we have an HMC, we won’t need any additional hardware on your machine”. I told him that iTech Solutions can do this in one day for him. Santa was like a little kid at Christmas, and said, “Well that is another thing I want from iTech Solutions”.
Thirdly, he asked if there was anything we could do, as IBM had assigned him fifteen different customer numbers for the North Pole, each reindeer had their own IBM customer number, and a few of the elves had IBM customer numbers as well. Santa said, “It’s confusing when we have to call IBM, and they ask us for a customer number. I don’t know which one to give him.” I laughed, and said, “Sorry, even iTech Solutions can’t help untangle IBM customer numbers”. I think you could hear his belly laugh all the way down at the South Pole.
We were just about at the end of the day, and it was so quiet in the computer room, not a creature was stirring, not even a mouse. Santa whispered in my ear, that he was so happy to have iTech Solutions on his team, that he was making sure we were all on the Nice list. I in turn asked Santa, “What are the people on the Naughty List getting for Christmas besides Coal?”. He said, “Because they have been so bad, I am giving them Windows Servers. After all the crashes, problems, and headaches they will have with those Windows Servers, they will think twice about being naughty again.” I nodded my head in agreement.
Then Santa said, “You know the last time you were here you told me that I should back up the system and move my tapes to the South Pole. Well, I am a little concerned that if we lost our tapes, then everyone would know who is on the Naughty and Nice list.” I said, “Not to worry, you have two options. We can do software encryption with BRMS, or we can do Hardware Encryption with that fancy new Fibre attached LTO-7 tape drive we installed last spring, or the SAS attached LTO-7 tape drive on the new 9105-42A.” I could see Santa was very happy as there was a sparkle in his eye. We planned to do the hardware-based Tape Encryption using SKLM in February. If your tapes are not encrypted what would happen if you lost a tape? Who would have your data? Let iTech Solutions Encrypt your tapes.
As we wrapped up our year with Yvonne leading our last monthly MSP meeting we looked over all the projects we completed this year. Santa reported that his elves had so much more time to implement new applications in modern languages as iTech handles all the system administration needed for his IBM i OS and Power hardware. Further, he has the peace of mind that comes from partnering with a trusted team of highly competent specialists. We are excited to look to what we will accomplish together in 2023!
If Santa wants iTech Solutions for his support, then shouldn’t you?
So, I am glad to report that iTech Solutions is what Santa wants for Christmas.
This newsletter includes: