SPS Technology

WiT - Giving Feedback with Lisa Even

Wed, 16 Aug 2023 12:21:59 -0500

Women in Tech organized an event on August 1, 2023 for folks to network and learn from the external speaker Lisa Even. Lisa had a dynamic and interactive presentation about how to give productive and useful feedback. These are some of my key take-aways from the session:

Feedback Triggers: These are triggers that could make someone respond negatively to feedback. When giving feedback, we must be aware of these and respond appropriately.

Truth: Does the person believe this person and their feedback?
Relationship: How much does this person trust you? Do you have a good relationship?
Identity: Does this feedback challenge someone’s identity or bring up an insecurity?

Types of Feedback: It’s important to know the type of feedback someone is looking for or identify the type of feedback you want to give to someone to communicate it effectively.

Appreciation: Giving people kudos for their work. It’s important to be timely, authentic, specific, and give your full attention to this feedback. It should happen in an intentional manner, not while multitasking or while distractions are present.
Coaching: Giving someone feedback on what they can do better.
Evaluation: Telling someone how they are doing compared to others around them. This type of feedback can be tricky to deliver in a way that is helpful.

Tips for Giving Feedback:

Listen first – people want to feel heard and on the same team as the person giving feedback.
Lead conversations with small talk to peek into their wellbeing.
Ask Questions:
- How did you think this went?
- What felt hard or easy about this task?
- On a scale of 1-5, how much did you like this task?
- What do you think we should do?

Overall, the key takeaway in giving feedback is to listen more than you talk. This was a great event held by Women in Tech and I’m hoping that they host more speakers in the future.

Sarah Sullivan, Tech Trainer

Giving back to the AWS community

Mon, 23 Jan 2023 11:24:09 -0500

Open source has become the de facto way to build software not only in tech, but across diverse industries. As companies use open source code to build their own commercial products and services, they also see the strategic value of contributing back to those projects. Since at least 2017 SPS has seen the benefits of contributing back to the open source community in ways large and small. Recently with the acceptance of Add ClientMetadata to SRP auth flow #90 to the aws-sdk-net-extensions-cognito repository another step has been made to contribute back to the community and share additional features out to all consumers of this extension.

The .NET extension for Cognito supports developers using the C# programming language who also use the AWS Cognito service to easily make calls to the API. The repository puts it best:

Amazon.Extensions.CognitoAuthentication simplifies the authentication process of Amazon Cognito User Pools for .NET developers.

It allows you to use various authentication methods for Amazon Cognito User Pools with only a few short method calls, and makes the process intuitive.

One aspect of calling Cognito which wasn’t covered in the extension was the ability to provide Client Metadata to a custom token generation lambda function. Because the lambda token generation function needs to have a very short runtime it is inadvisable to reach out to a database in order to add custom scopes or other information. Additionally, some information may be sensitive or not stored in a database and can or should only be passed by an API call. For this and other reasons the Cognito API provides a Client Metadata object as part of the Initiate Auth request.

For some software used at SPS, information which was available in the API needed to be passed to Initiate Auth and consumed in the pre-token generation lambda to be sent to the end user. A pull request was opened and accepted, so now as of Amazon.Extensions.CognitoAuthentication 2.2.4 it is possible to pass “ClientMetadata” along.

We hope that this contribution will be of great assistance to the community of C# Cognito users and enable more users of the Cognito service to take advantage of customizing their Cognito tokens!

-Jonathan DeMarks, Principal Cloud Engineer

SPS Partners with, and Sponsors, Society of Women Engineers - University of MN Chapter

Tue, 18 Oct 2022 13:30:36 -0500

SPS Commerce is excited to partner with and sponsor the University of Minnesota Chapter of SWE (Society of Women Engineers) for the Fall 2022 – Spring 2023 school year!

Throughout the year, there will be multiple opportunities for SPS to get involved and support SWE’s mission to empower women to achieve full potential in careers as engineers and leaders, expand the image of the engineering and technology professions as a positive force in improving the quality of life, and demonstrate the value of diversity and inclusion. Stay tuned for updates on how we’re supporting women of all ages in STEM education by building strong connections with technical women leaders.

Interested in learning more about SWE Minnesota? Take a look at their website!

Lorrin Murphy

Wed, 12 Oct 2022 13:57:06 -0500

My Journey to Becoming an SPS Intern (+Tips and Tricks)

This post is the first part of a series in which we explore the journey to becoming an SPS Intern along with Melissa Wood, 2022 Summer Technology Learning Services Intern at SPS Commerce.

Post 1: Applying

When I was applying for internships last fall, there was a lot about the process that I was unaware of. I thought it would be easy and that I would land a role quickly (boy, was I wrong). After around 70 applications, ten interviews, and a few job offers, I landed my internship at SPS Commerce. In this blog post series, I will be going over my experience. My hope is by breaking down the process, I can help you be successful in your search for the perfect internship.

Before I begin, I want to give some context: Last fall I was a third-year student at the University of Minnesota-Twin Cities majoring in Human Resource Development. Most students believe that the summer of your junior year is THE TIME to land your “perfect” internship to gain experience in the field you want to pursue post-grad. Obviously, this is not true, and it leads to students putting unnecessary pressure on themselves (more on that in a later blog post). I knew I wanted to work in the learning and development field. I also was particularly interested in pursuing internships at tech companies. With this knowledge, it was time to get started on the process of finding an internship.

Prior to applying to internships, I spent time updating my resume. This included updating to show new roles, changing language to action verbs, and adding resume keywords. I also visited my university’s career office, which was super helpful. Their feedback was to tailor my resume to the job posting I was applying for. Although it was time-consuming, it allows recruiters to see that you have the necessary qualifications for the role. I also recommend (kindly) asking individuals in your field or peers if they would be willing to look at your resume, as it is only natural you will miss things after spending so much time editing and reviewing it.

After updating your resume, it is time to send out applications. This is something that I initially struggled with. I would be super motivated for one week and send out five applications and then for the following two weeks send out none. Eventually, I started to time block an hour or two each week to make sure I was applying to roles every week. I focused on finding roles through job boards such as LinkedIn, Indeed, and Handshake, but depending on your field there may be specialized job boards that post internship opportunities.

In the end, internship applications are somewhat of a numbers game. It is important to apply to many internships, even if you do not meet all the qualifications. For example, I did not have the experience with all the digital technology tools requested in the job posting, but I met the rest of the qualifications. Sometimes you just have to shoot your shot and apply anyways.

When applying to roles, it is also important to track your applications. I set up a basic excel spreadsheet that included the company’s name, the role I applied for (and a screenshot of the job posting to reference if I was moved on to interviews,) the status of my application and interview dates. This helped me stay organized and allowed me to track where I was in the application process.

Alright, so you have learned about updating your resume, sending out applications, and tracking applications. Now what? Instead of waiting for the magic to happen, it is a good idea to connect with individuals working at the organizations you are interested in. In the next post, I will be covering the importance of career fairs and networking when searching for an internship.

Series Author: Melissa Wood, Technology Learning Services Intern

Celebrating Hacktoberfest as SPS

Mon, 10 Oct 2022 11:22:19 -0500

Autumn is my favorite season in Minnesota. Something about the cooler temps and crisp air, the leaves changing color and falling from the trees, giving the air a familiar smell, and a crunch underfoot when heading out for a brisk walk. One of my favorite things about October in particular is that it is the home to Hacktoberfest, an event that encourages everyone to participate in open-source! Whether you’re a veteran programmer, or brand new to coding, Hacktoberfest offers a great opportunity to connect with open-source projects you know and use, explore new projects, add new features or enhancements, squash bugs, fix typos, clarify documentation, or draw that diagram that really helps a project community understand what the project’s goals are!

Several members of Team SPS Tech are actively involved with Hacktoberfest. We’re sponsoring and hosting Hacktoberfest Hacking Session (in-person and virtual) with GoMN. We have a couple open-source projects we are welcoming the Hacktoberfest community to participate and collaborate on:

REST API Standards & Design Style Guide
nap, the tiniest “sleep” program that fits into a Kubernetes ConfigMap

Whatever your October holds, we hope you enjoy this season, take a moment to appreciate family, friends, and also open-source!

– Jesse Lang | Principal Site Reliability Engineer | Technology Operations Engineering

DevEx-Powered API-led Product Growth

Thu, 06 Oct 2022 09:49:51 -0500

With the release of our REST API Standards, SPS Commerce is committed to continually evolving our Developer Experience through APIs and beyond. Join SPS Commerce and the folks at Tyk.io, on the All About APIs Podcast by Tyk as we dive into Developer Experience and API-led product growth.

Ways to listen: - Apple Podcasts - Spotify - Stitcher

Episode #002 - SPS Commerce’s Travis Gosselin talks DevEx-powered API-led product growth

Welcome back to episode #2 of the All About APIs podcast. This week, we talk all things DevEx with SPS Commerce’s Principal Software Engineer, Travis Gosselin. DevEx. What is it and what is its place in the realm of API-led product growth? Take a listen to learn more, and let us know what you think!

-Travis Gosselin, Principal Engineer, Developer Experience

Reference: ….http://blog.travisgosselin.com/devex-powered-api-led-product-growth/

September GoMN Meetup

Fri, 02 Sep 2022 12:55:16 -0500

Real people and real pizza, what a combo! Join us in person or virtually for “Observing Golang: OpenTelemetry from Code to Prod”, presented by Roni Dover, CTO and Co-Founder at Digma.

GoMN Meetup, September 21, 6pm CDT at SPS Tower

We also provide a virtual option, via Zoom, for those uncomfortable attending in-person.

Meetup Registration Here!

SPS Commerce joins TechNova Women+ in Tech Hackathon

Tue, 30 Aug 2022 14:55:52 -0500

SPS Application Engineer Charlotte Countryman shared her experience as a judge for the TechNova Hackathon! SPS Commerce was proud to team up to support the student organizers from the University of Waterloo with their second Women+ in Tech Hackathon. Read about the two projects that stood out the most to her over here.

SPS Commerce REST API Standards Now Open Source

Fri, 26 Aug 2022 12:41:38 -0500

The SPS Commerce Tech Team has been working tirelessly over the past few years in curating a single set of technology best practices that we call Guardrails. These Guardrails are the culmination of years of experience from our engineering teams in designing, building, implementing, deploying, and operating best-of-breed software on the world’s largest retail network. These Guardrails consist of high-level technology decisions such as Language and Framework selection or DevOps Practices and drive all the way into expectations around Secret Management and Environments. As you would imagine, it’s not easy to aggregate the knowledge and water-cooler discussions from hundreds of engineers across multiple continents. The most important aspect of that is achieving singular alignment and agreement of approaches that can be codified into SPS’ internal engineering platform we call Atlas, which is where SPS Commerce is building its next-generation API Platform. On top of both of these initiatives has been the foundational requirement to pursue and dive head-first into API-First culture and align SPS Product, Technology, Processes, and People with clear API Standards and Guidelines for approaching REST-style APIs. SPS Commerce’s first iteration of REST API Standards (also known as a design guide or style guide) is now available publicly and open source.

Innovation is for Everyone!

Mon, 01 Aug 2022 12:47:14 -0500

Recently, I joined the first hackathon, aka innovation project, of my life. The hackathon was part of our internal Tech Jam conference at SPS, where we work on solving challenges, exploring ideas, learning new things, building, and sharing our wins and findings with the entire tech team to close out the conference. I had always hesitated to be part of a hackathon event. I thought this was only for software engineers, programmers, coders, and developers. I do not have a technical background so I was unsure what the expectation was and worried about whether or not I would fail or mess up. I was wrong! Innovation is for everyone.

Thank you to my leaders who pushed me and challenged me to try new things. In the last 3 days of the hackathon, I learned a lot from working with my incredibly talented team members while completing our innovation project.

Our hackathon project aimed to discover learning data to help analyze learning experience, learner frustration, and engagement in online training. We wanted to explore and research which capabilities and features are available in Microsoft Clarity and Google Analytics to track users’ experience with the Rise 360 authoring tool. Some of the preliminary questions we had in our mind were:

Can we get the data from Rise 360 to Microsoft Clarity or Google Analytics?
Can we get data on which resources or topics learners look for most?
Can we get data on how users interact with the course? Where do they spend most of their time? How much time do the users spend on a course? How many times do they click during the course?

With this data, we could track, analyze, and report the effectiveness of an online training course. We could measure and understand how our learners interact with the online course and dig in to understand learners’ frustration. It gives us insights into what learners are doing and strategies we should apply to improve or minimize this behavior.

My biggest takeaway from this experience is that failure is okay and that your project does not have to be perfect. In fact, you don’t have to be perfect. As Reshma Saujani stated, “Brave, Not Perfect is about understanding how we got here so we can navigate our way out.” Being part of a diverse workforce of technical and non-technical teammates working together on our innovation project was amazing. We were unstoppable and did not give up. I noticed we all shared a natural curiosity and a thirst for a growth mindset (one of the core values at SPS). Our curiosity raised questions like: What if we try this? What if we do this? thus paving the path for our innovation project.

I was super pumped even with our small successes, which gave me hope and allowed me to go deeper and explore more questions with my curiosity. Our team was open to suggestions when innovating and did not follow one strict way of exploring. I remember a light bulb moment for our team members when we discovered a feature that allowed us to get the valuable data we were hoping for. This discovery fueled us with excitement and joy. 😊

Collaborating with team members energized me and expanded my comfort zone. This experience has made me more open and curious for suggestions, exploring, and trying things to see how they fit in. I now view hackathons as being not just for technical people but for everyone. Everyone is capable of being curious and we all have the spirit of a child inside of us which can lead us to be resilient and to learn and grow for better outcomes. I feel I had a huge success with my first innovation/hackathon project. An additional perk is that I not only got to work on a new product but am also able to expand my network, which was a huge win in itself. You never know what signing up for a hackathon or pursuing innovative work will lead to. So, what are you waiting for?

Anu Joshi, Sr. Technical Trainer

Reference: Saujani, R. (2019). Brave, Not Perfect: Fear Less, Fail More, and Live Bolder. United States: Currency.

TechJam 2022: Back to the Jam!

Thu, 21 Jul 2022 13:44:28 -0500

Each year SPS Technologists participate in a volunteer-led internal tech conference, called TechJam. Read more about a few past TechJams from 2020 & 2021! 🎉 Our team of 27+ volunteers did everything to get TechJam up and running this year: from hosting Zoom prep sessions to crafting NFTs, from running a CFP process to source talks to creating our awesome logos. As the week wraps up, we wanted to share more about the process of TechJam.

TechJam Goals

This is our 7th year of hosting an internal conference, and we’ve gotten pretty good at articulating our ‘why’ which drives how we make the magic happen. TechJam has three primary goals:

to embrace learning (All great tech organizations are learning organizations!)
to connect our teams with each other and to the SPS Tech Community
to encourage innovative thinking, and approach solutions with a broad and expansive set of techniques and experiences.

So Many Speakers!

Speakers at TechJam are volunteers, members of our teams who want to share things they are exploring, have already mastered, or want to discuss with peers. Talks are organized across 3 concurrent tracks, presented virtually to our teams aroung the globe. 🌍 With over 885 minutes of content, it’s hard to pick some to highlight, but here is a small list of some talks:

Lessons we Learned from Running a Hybrid Infrastructure
Using Gamification to Learn Programming Languages
Building Custom Cars & Software Engineering are not as Different as you Think!
Accelerating Developer Experience with API Design First
Imposters Among Us
Making Life Easier for Developers with AWS SSO
Stretchhhhh Yourself
How I Learned to Stop Worrying (about CRUD) and Love GraphQL
The Endless Possibilities of NFTs

We also welcome external speakers to TechJam, voted on by our organizers and selected for the exciting topics they share. This year’s presenters spoke on the value of continuous delivery and responsible AI development.

Innovation at TechJam

We believe in fostering a spirit and culture of innovation at all times and we intentionally set aside time during our Tech Jam event for people to … you guessed it, innovate. 😉 There were 20 teams participating in our innovation event and showcasing the results of their work together as the capstone of our event. The three types of innovation groups this year were: Hackathon, Education & Book Club (defined by the participants and their innovation interests). Innovation HighlightsEvent POAPs!

Event POAPs!

As if an abundance of amazing content, innovation highlights, and general fun with our peers weren’t enough, we also provided an official ‘Back to the Jam’ POAP to all attendees, speakers and event organizers. We see you, Web3.

See you next year, TechJam!

eBook: The API Roadmap

Tue, 05 Jul 2022 14:32:47 -0500

Stoplight recently made available a free ebook resource titled The API Roadmap: Secrets to API Strategy Success from Industry Leaders. This is an amazing resource that pulls together the information and insights from the series of API Intersection podcasts that have taken place over the last year and beyond. If you are building or starting an API-First initiative or program this is a MUST read!

SPS Commerce had the opportunity to join the API Intersection podcast a few months back and share our journey so far into API-First culture. You can also find an excerpt of that journey in this publication.

-Travis Gosselin, Principal Engineer, Developer Experience

Reference: http://blog.travisgosselin.com/ebook-the-api….

How we used Jaeger to improve the performance of an application

Fri, 10 Jun 2022 10:26:17 -0500

At SPS Commerce we are moving all our APIs into Kubernetes. There are many reasons for this change, but one of the more fun reasons is the ability to use a tool like Jaeger. Jaeger is a distributed tracing tool, useful for seeing how your APIs and services all work together to process one request.

A lot of the initial setup of this infrastructure was performed by our wonderful SRE team. As an application team that maintains a Django REST Framework application, there wasn’t a lot of additional work required to start seeing the advantages of tracing. We added and configured the OpenTelemetry Django, Requests, and Psychopg2 instrumentors to generate spans representing different operations within our application. These instrumentors capture interactions between our application and its dependencies, and they plug into the application with very little changes or configuration required. Then we configured a Zipkins exporter to send those spans to our platform’s shared collector.

After opening the Jaeger UI and filtering down to only traces including our application, we started browsing through the traces, just exploring the tool. Very quickly a few endpoints began to stand out due to their high response times and high number of spans required to complete a single request. Here is an example trace of one of the problematic endpoints, drilled down into the database spans:

This is a complicated endpoint, joining together data from multiple tables, but no endpoint should be making this many calls to the database to generate a single response. As most Django developers have likely found out: inefficient use of the Django ORM can have catastrophic effects. A brief investigation uncovered code that iterated through a queryset, and for each entry in the queryset it made an individual call to the database to retrieve a different object. Then to make matters worse, those objects were then passed to a serializer that included a nested serializer for one of their related objects! The result was for users who had associations with many of these objects, or with objects that were particularly complex or nested, a single call to this endpoint could generate more than one hundred calls to the database!

By using the very helpful django_assert_num_queries pytest fixture, we were able to write a test that measured the number of queries this endpoint made and could easily reproduce the issue. After some simple refactoring and a well-placed call to prefetch_related_objects(), we were able to reduce that endpoint down to a manageable handful of queries. The drop in response times after we deployed the fix pretty clearly reflect the improvement:

We have only just begun to scratch the surface of what we can do with Jaeger. But already, the introspection and visibility it provides have been incredibly helpful. As more and more of our APIs migrate onto our Kuberenetes platform, the benefits will only grow.

Matt Kokotovich, Principal Engineer

New WiT Steering Committee

Mon, 23 May 2022 16:01:00 -0500

SPS WiT is proud to announce a new steering committee! Thank you to outgoing members, Annie Greenawalt and Megan Tischler for their leadership. It has been exciting to see your passion shine during your time in this role. Congratulations and welcome to the next generation of leaders! We can’t wait to see what you bring to the team.

SPS Hosts GoMN this Wednesday, May 18 at 6pm

Mon, 16 May 2022 08:26:14 -0500

This Wednesday, May 18, at 6:00 pm, the GoMN Meetup is welcoming John Murphy who will present Debugging Go Programs in Kubernetes. A brief summary of the topic can be found here, where you can also register to attend!

The event is sponsored by SPS Commerce, is BOTH in-person AND virtual, and everyone is welcome to join - this is a free event :).

SPS Commerce + No Fluff Just Stuff (NFJS)

Wed, 11 May 2022 09:01:04 -0500

If you have ever gone browsing for some great conferences for software engineers, you likely have come across No Fluff Just Stuff, aka NFJS. NFJS offers a network of conferences and webinar topics, both big and small concentrating on anything software developer and architecture focused. They offer content weekly in the form of both free webinars and full-day workshops across a variety of topics. Content is delivered by experienced architects and engineers with first-hand expertise in the topical area. Larger conferences also materialize multiple times a year with ArchConf and UberConf, which offer a mostly vendor-free experience with in-depth 90-minute sessions.

I was able to first attend ArchConf in 2019 and immediately fell in love with the longer sessions that allow you to dive deeper and get beyond some of the surface-level content. Each day is jam-packed full from early morning to well after dinner in the evening in a very accessible location where you are not walking 20 min to your next session.

Earlier this year I had the fantastic opportunity to join the schedule with some of the NFJS tour stops and represent SPS Commerce with some of our latest development and patterns and practices. The opportunity to share with the external community as well as learn from the audience and engage with engineers across the industry is incredibly valuable.

The first stop was an NFJS Free Webinar that took place on May 6, 2022, remotely. The webinars are zoom-based and allow for great bi-directional discussion. Had some great questions early on and good engagement throughout the hour. Feel free to catch the recording if you are interested where we dive into the basics of Feature Flagging: Amp Up Deploy Velocity with Feature Flags.

Coming up, you can also find me in person, representing SPS Commerce, with further topics at UberConf and ArchConf.

Compelling Code Reuse in the Enterprise

Sharing code and internal libraries across your distributed microservice ecosystem feels like a recipe for disaster! After all, you have always been told and likely witnessed how this type of coupling can add a lot of friction to a world that is built for high velocity. But I’m also willing to bet you have experienced the opposite side effects of dealing with dozens of services that have had the same chunks of code copied and pasted over and over again, and now you need to make a standardized, simple header change to all services across your platform; talk about tedious, frictional, error-prone work that you probably will not do! Using a variety of code-sharing processes and techniques like inner sourcing, module design, automated updates, and service templates, reusing code in your organization can be built as an asset rather than a liability.

In this talk, we will explore the architectural myth in microservices that you should NEVER share any code and explore the dos and don’ts of the types of reuse that you want to achieve through appropriate coupling. We will examine effective reuse patterns, including what a Service Template architecture looks like, while also spending time on the lifecycle of shared code and practically rolling it out to your services. We will finish it off with some considerations and struggles you are likely to run into introducing code reuse patterns into the enterprise.

Accelerating Developer Experience with API Design First

Modern HTTP APIs practically run the contemporary tech world. The number of APIs your organization is actively building and maintaining is evidence of that, and you need no convincing of the value of API Design First principles. However, introducing an API Design First process and methodologies can be fraught with too much manual effort, slow progress, inconsistencies, and further chaos as you’re organization scales. Much of this friction can be alleviated by developing a mature API Design First process within the organization that is supported by first-class tooling and the use of automation.

In this talk, we will dive into the principle areas of API Design First across its lifecycle as we discuss how to accelerate value in design, development, governance, documentation, and change. Whether you already have established API Design First methodologies or are considering how to effectively adopt it, you should leave with a practical understanding of effective processes and governance. Experience how SPS Commerce thinks about API Design First with a strong preference towards governance through collaboration along with examples of key processes that simply must be automated to succeed in an API-First world.

Unleashing Deploy Velocity with Feature Flags

A lot of development teams have built out fully automated CI/CD pipelines to deliver code to production fast! Then you quickly discover that the new bottleneck in delivering features is their existence in long-lived feature branches and no true CI is actually happening. This problem compounds as you start spinning up microservices and building features across your multi-repo architecture and coordinating some ultra-fancy release schedule so it all deploys together. Feature flags provide you the mechanism to reclaim control of the release of your features and get back to short-lived branches with true CI. However, what you’re not told about feature flags in those simple “if/else” getting started demos is that there is an upfront cost to your development time, additional complexities, and some pitfalls to be careful of as you begin expanding feature flag usage to the organization. If you know how to navigate these complexities you will start to unleash true velocity across your teams.

In this talk, we’ll get started with some of the feature flagging basics before quickly moving into some practical feature flagging examples that demonstrate its usage beyond the basic scenarios as we talk about UI, API, operations, migrations, and experimentation. We will explore some of the hard questions around “architecting feature flags” for your organization.

-Travis Gosselin, Principal Engineer, Developer Experience

Reference: http://blog.travisgosselin.com/sps-commerce-nfjs…

AWS SDK Credential Provider Multi-Region Behaviors

Wed, 27 Apr 2022 15:35:42 -0500

In the course of investigating an issue with a multi-region deployment in AWS EKS, I ran into not just some obscure behavior from the application operating in the second region, but just straight-up bugs in the AWS SDK for .NET itself. Investigating further across other SDKs in different ecosystems and there are interesting behaviors for region configuration across them all; both good and bad.

Problem

It started on the first deployment of a standard .NET Core containerized app to SPS Commerce’s internal multi-region compute platform based on AWS EKS. I was simply adding some resiliency to my application that had been running in us-east-1 for a long time to now shift some of its traffic to the same app running in us-east-2. Typically this transition is pretty straightforward with the internal deployment mechanisms and infrastructure provided by SPS. This time it started failing with the following behaviors:

The application would fail on start in the second region only (i.e. us-east-2).
There was zero hardcoded configuration for AWS regions in the codebase and the app was driven by region using the “AWS_DEFAULT” environment variable specified during configuration in each respective region.
Secrets from the AWS Secret Manager are loaded and merged with the startup configuration very early in the pipeline.
Subsequent services are used in AWS depending on the API request coming in, including SQS and Dynamo.

The initial symptom of failure during deployment to us-east-2 was the Kubernetes readiness probe failing to successfully ping the APIs endpoint at “/healthz”. This was because the container was not starting and the only log output was “STARTING-HOST”. No further logs were provided. The readiness check would eventually fail entirely and stop the container, and startup up another with the same result.

Through more verbose logging enablement with Serilog in ASP.NET Core, it was obvious that loading secrets from AWS Secret Manager early on in the startup pipeline were stuck in an endless loop attempting to access AWS STS service to grab credentials to make the AWS Secret Manager call (provided via OIDC with WebIdentity in EKS).

Take a look at the more verbose logging output:

AmazonSecretsManagerClient 28|2022-04-18T13:38:26.824Z|INFO|AmazonClientException making request GetSecretValueRequest to [secretsmanager.us-east-2.amazonaws.com/.](https://secretsmanager.us-east-2.amazonaws.com/.) Attempting retry 1 of 4.
DefaultConfigurationProvider 29|2022-04-18T13:38:27.226Z|INFO|Resolved DefaultConfigurationMode for RegionEndpoint [us-east-1] to [Legacy].
AmazonSecurityTokenServiceClient 30|2022-04-18T13:38:37.232Z|ERROR|An exception of type IOException was handled in ErrorHandler. --> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   --- End of inner exception stack trace ---
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
AmazonSecurityTokenServiceClient 31|2022-04-18T13:38:37.233Z|INFO|IOException making request AssumeRoleWithWebIdentityRequest to [sts.us-east-1.amazonaws.com/.]([sts.us-east-1.amazonaws.com](https://sts.us-east-1.amazonaws.com)/.) Attempting retry 1 of 4.

Notice that: - It wants to use us-east-2 for requesting the secret. - It then seems to switch back to us-east-1. - Attempts to call AWS STS in us-east-1 instead of the us-east-2 region: sts.us-east-1.amazonaws.com - No further error logging results, since the retry is hidden information unless you configure it to expose it for debugging. - This loop continues endlessly until the container orchestrator eventually gives up.

Of course, I scour the code looking for possible references or indirect references to hardcoded regions. I did not find it. Shelling into the container in the cluster, I discover I can use AWS-CLI just fine and it works and respects the AWS_REGION environment vars. This tells me the platform and infrastructure are configured fine.

Consultation with our platform team leads me into examining: AWS_STS_REGIONAL_ENDPOINTS environment variable. This seems directly related to what I’m experiencing. This environment variable defines what region and behavior the AWS STS initial authorization request will use and how that might be different from the AWS_REGION environment configuration. It basically boils down to “legacy” (which uses a global endpoint), and “regional” which should just use AWS_REGION environment variable essentially. I had thought for some reason my app was attempting to use the global endpoint, which depending on the region might redirect to use AWS STS in us-east-1 by default. I confirmed it was set to regional and working properly inside the deployed application.

Diving into the AWS SDK for .NET

The investigation lead me into reviewing the AWS SDK for .NET to identify how the region materialized in the cases above.

I updated my demo app to be a single AWS STS call with no dependency injection and a straight-up hardcoded region for simplicity and clarity.

using (var sts = new AmazonSecurityTokenServiceClient(RegionEndpoint.USEast2))
{
    var result = await sts.GetCallerIdentityAsync(new Amazon.SecurityToken.Model.GetCallerIdentityRequest());
    var userId = result.UserId;
}

It appears that the Constructor for service instantiation in the AWS SDK properly passes the region along to the config. However, digging deeper, that region passed in is used for the actual request, but to grab the initial identity credentials (requested anonymously) it creates an AWS STS client in which it re-discovers the region:

var configuredRegion = AWSConfigs.AWSRegion;
var region = string.IsNullOrEmpty(configuredRegion) ? _defaultSTSClientRegion : RegionEndpoint.GetBySystemName(configuredRegion);

In this case, the default STS Client Region is hardcoded as us-east-1:

private static readonly RegionEndpoint _defaultSTSClientRegion = RegionEndpoint.USEast1;

Based on this behavior we can identify this is likely the reason that we are getting redirected to us-east-1 for our role assumption. But why didn’t the value from AWSConfigs.AWSRegion use the region value that we configured? Largely because AWSConfigs does not use any provided region fallback chain or lookup based on the AWS_REGION environment variable. It simply checks existing local configuration files for it. Additionally, this implies that the environment variable AWS_STS_REGIONAL_ENDPOINTS really only works in the SDK for subsequent calls after credentials are retrieved anonymously.

This behavior exists int credential sources for AssumeRoleWithWebIdentityCredentials, AssumeRoleAWSCredentials and CognitoAWSCredentials.

In good fashion, my investigation led me finally to an existing GitHub Issue (AWS_STS_REGIONAL_ENDPOINTS as environment variables) for this exact problem: (wish I was able to find this sooner). Unfortunately, this issue has been active for almost a year by this point without a formal fix.

AWS SDK for .NET Workaround

Based on the investigation above we know that we can easily work around this problem by setting the AWSConfigs.AWSRegion value before the AWS SDK initializes, in this case before secret manager requests go out on startup in our startup middleware.

// bootstrapping configuration
var builder = WebApplication.CreateBuilder(args);

// MUST manually set region to force us-east-2 in deployed app
// otherwise falls back to us-east-1 STS in all regions in SDK
AWSConfigs.AWSRegion = Environment.GetEnvironmentVariable("AWS_REGION");

This will ensure that the value used by the initial role assumption AWS STS calls use the same value as provided in AWS_REGION environment variable. This means that AWS STS calls will be mimicking the “regional” setting of the value AWS_STS_REGIONAL_ENDPOINTS, which is the recommended non-legacy approach anyhow, and exactly what we need here.

It would definitely be great to contribute some enhancements to the AWS SDK for this. Time allowing that would be great, but contributions with the code in a local runnable state and proper testing I expect is not trivial, especially across all affected credential providers. Maybe you have some time?

Behaviors in Other AWS SDKs

Given the polyglot ecosystem we have across our applications at SPS Commerce I was interested to see how this behavior looked in other AWS SDKs we might have teams using.

AWS SDK for Java V2: functionality is definitely different but closer to what you expect where the credentials are retrieved using the DefaultAwsRegionProviderChain which does in fact include pulling the region from AWS_REGION environment variable; essentially what our shim/workaround for .NET does above, but just natively through the existing region provider chain. This is how I would expect the .NET AWS SDK to be updated. This effectively makes the Java SDK act as if it was set for “regional” STS endpoints. Although interesting, the Java SDK makes no usage or reference to the AWS_STS_REGIONAL_ENDPOINTS environment variable for legacy support, it does fallback to the global STS endpoint when no region is provided.

AWS SDK for Python (Botocore) - functionality appeared to work as you would expect by passing along the configured region for the service client your instantiating, including support for AWS_STS_REGIONAL_ENDPOINTS environment variable. However, interesting to note is that the region used is always cached as the supplied first region, and cannot be changed with subsequent client requests when retrieving credentials. The provided note indicates that IAM/STS is really global, so credentials retrieved by one region-specific endpoint are valid in another region. Worth noting that might seem odd in your application logs to continually see STS credentials garnered from one region for applications that may be accessing multiple regions with different clients. I’m not settled on if this poses potential resiliency issues if the “first cache” regional endpoint for STS becomes unavailable and you are expecting your application to be able to efficiently switch over to a different region. What do you think?

-Travis Gosselin, Principal Engineer, Developer Experience

Reference: http://blog.travisgosselin.com/aws-sdk-credential…

SPS Hosts GoMN this Wednesday, April 20 at 6pm

Mon, 18 Apr 2022 12:19:18 -0500

This Wednesday, April 20, at 6:00 pm, the GoMN Meetup is welcoming Raj Auradkar who will present Dependency Injection and Mock-Based Testing. A brief summary of the topic can be found here, where you can also register to attend!

The event is sponsored by SPS Commerce, is BOTH in-person AND virtual, and everyone is welcome to join - this is a free event :).

SPS Commerce at the MN Tech Job & Opportunity Fair on 03/24/2022

Wed, 23 Mar 2022 12:34:00 -0500

SPS Commerce will participate in the MNtech Job and Opportunities Fair at The Machine Shop on Thursday, March 24th, 2022, at 530pm. Please, stop by our booth; we will be happy to make new friends and answer all your questions regarding our thriving technology business!

The event is free, with free food and drink! Click here to register!

Hope to see you there 😊

Developer Experience & API Style Guides at SPS Commerce

Fri, 04 Mar 2022 14:27:58 -0500

SPS Commerce is moving fast in adopting an API-First culture and mindset, as we re-envision our API Platform from the ground up. Some amazing work from our internal tech community and engineering teams over the last year. Dive in and check out our journey so far in the latest API Intersection podcast as we chat about API Style Guides and Developer Experience.

Shout out to the API Intersection Podcast team over at Stoplight who are producing fantastic content on a regular basis, that we have used to help guide our direction at SPS. It was a great opportunity to be able to share and discuss our learnings so far with the community. Looking forward to continuing to share the SPS Commerce + API-First journey and learning from the great technology community around us (find us at UberConf 2022).

Stoplight Blog Post & Summary: Collaboration for Successful Scaling: Developer Experience & Style Guides from the Ground Up with SPS Commerce

-Travis Gosselin, Principal Engineer, Developer Experience

Reference: http://blog.travisgosselin.com/developer-experience-api-style-guides…

SPS Commerce + Azure Pipelines: A CI/CD Journey

Fri, 28 Jan 2022 13:44:06 -0500

SPS Commerce is a “booming” technology and SaaS-based organization to work for. Like any enterprise that has consistently done double-digit growth for the last 20-years (I say that like there are a lot… but not as many as you might think), the technology and development-based initiatives on the rise make for an interesting eco-system of tooling that a developer navigates on any given day. That tooling can also be wildly different across different teams and departments. CI/CD tooling is no exception and is an incredibly large piece of day-to-day usage for many engineers. That experience alone can change the tone of your day, and also affect large aspects of where you spend development time and potentially undifferentiated engineering.

For many years at SPS, CI/CD tooling and the method of deployment were often compared to a “choose your own adventure” style of architecture. Plenty of good tooling and patterns were made available across many different generations and iterations of development at SPS. However, no single tool could get you from code to production. Some newer generations of components could get you to production with less tooling and fewer boundaries to cross with varying degrees of setup. Once a team had established “their” pipeline architecture of choice, it was then stamped out for all the services under that team’s responsibility. The diversity of tooling obviously meant it was difficult, if not impossible, to roll out some features and workflow orchestration updates across all pipelines or even a few pipelines. Teams needed to support at least some of their own infrastructure for customized build instances, and in some cases deploy infrastructure as well.

With the introduction of Azure DevOps, and more specifically Azure Pipelines, our workflow orchestration and standardized build and deploy capability is now available across the organization, across departments, and across cloud providers. Azure Pipelines has enabled SPS to centralize and architect reusable workflow templates that are generic enough for each team to use, while at the same time providing the capability to modify all pipelines workflows and incorporate security best practices in real-time updates. Centralized infrastructure for orchestration and rollout also leaves teams with management of zero infrastructure themselves. Centralized documentation and example patterns enable teams to get up and running quickly with usually a single way to do something. Team contributions to new features and capabilities within the platform mean that everyone can make use of new functionality almost right away.

As a Principal Engineer focused on Developer Experience, I recognize this type of improvement for our internal engineering teams has an extremely important and powerful step in our continued journey with CI/CD maturity. However, this important step didn’t happen overnight. In fact, it started almost 6 years ago…

Origin of Azure DevOps at SPS

Azure DevOps is an obvious suite of tools you might choose for those development teams and organizations focused largely on the Microsoft stack. You are likely deploying to Azure and making use of Azure DevOps cloud-hosted agents for Windows builds. As a team using Team Foundation Server (TFS), using Azure DevOps was an incredible upgrade with lots of potentials. This is exactly why I introduced it to our small development team in late 2015, and we began spiking and using its then “new” “Release UI” functionality in private preview. This was the first availability of Microsoft’s “InRelease” product purchase, now integrated into their platform. Not long after, that small development team was acquired by SPS Commerce. The beginning of 2016 put me and a bunch of others into an exciting, fresh, and well-funded development ecosystem that held best practices and security with very high regard. It also came with an ecosystem of other tooling that was now made available to us and our products.

Fresh off the heels of the excitement and feature capability of Azure DevOps, not long before the acquisition, we were excited to see what SPS Commerce had for CI/CD that we could make use of. At that time there were some good patterns in development that had good potential. The reality at the time was that each team was largely responsible for their own build and CI capability. Deployment had some centralization through a valuable abstraction called “BDP Core”, but centralized orchestration of that was limited in its capability compared to some of the features becoming available in GA of Azure DevOps. Additionally, no support or readily available patterns at SPS were available for non-Linux-based development either, to which cross-platform capability was a definite need.

As a result, the decision was made for that one team to move ahead using Azure Pipelines Builds + Release functionality to orchestrate builds and deploys to both on-premise and AWS.

The Early Days

As with the early days of most development, it was very exciting and fast-paced with our development of new pipelines and capabilities within Azure DevOps. A large part of that was due to the rapid development of the Azure DevOps platform at the time, with regular releases and features pretty much coming every two weeks as part of their internal sprint cycle at Microsoft. Externally, the Marketplace was growing fast as well. The combination of community-driven extensions and our own custom-built tasks enabled high velocity to quickly and easily deploy to many types of release targets.

Within the first year and a half, we found Azure DevOps growing to support not just a single team, but multiple teams within the Analytics department at SPS, with a wide variety of deployment patterns, including AWS deploys with Cloud Formation / Elastic Beanstalk / CodeDeploy, on-premise SQL Server deploys with SSDT and SSIS and Microsoft IIS WebDeploy. The advantage in this ecosystem was that most teams needing these patterns did not have to go back to a centralized group or individual but were empowered through the flexibility of Azure DevOps to use the centralized build and deploy agents with their own or community tasks. The built-in orchestration and workflow capabilities were already incredibly powerful and enabled you to design your workflow within a few clicks. This meant the capability and features of the Azure DevOps in-house patterns continually were up to par, if not ahead of some of the capabilities of the other CI/CD tooling within the SPS ecosystem, and that didn’t go unnoticed. Small development efforts additionally began integrating Azure DevOps with functionality like events and auditing directly into SPS’s centralized “BDP Event Stream” in AWS Kinesis to begin reporting the activity and traffic of Azure DevOps alongside the rest of the organization. This included information in the BDP Event Stream such as build time, duration, unit tests coverage as a first for aggregatable information about our build processes internally.

However, not all forms of development or all patterns being stamped out in Azure DevOps were necessarily part of an ideal future-proof architecture. As SPS Commerce was rolling out its highly advantageous containerized platform for runtime on top of AWS ECS, it was unclear how Azure Pipelines could effectively integrate with that. Security, best practices, and consistency were all top of mind on how CI/CD tooling would integrate with and deploy to the centralized container clusters for web apps and APIs. The “BDP Core” abstraction was expanded to support REST API interaction. With a well-designed custom task for BDP Core integration, Azure Pipelines was able to take the best of both worlds. It was able to use all the orchestration features of Azure Pipelines and reduce its responsibility on “how” it would push secure deployments to the cluster by using BDP Core API directly. A single tool producing builds and releases, with all deployments visible across each environment was still only achievable with Azure DevOps as a single suite. BDP Core API would then handle the conventions of how it would compose and push containers to the container runtime, and it could handle this centrally across all CI/CD tooling at SPS Commerce. This was a powerful win for CI/CD capability at the time; some great stories and learnings here for a future post from SPS Commerce.

Expanding Success

Our usage of Azure Pipelines began growing to more teams within SPS. With that, the Azure Pipelines patterns and infrastructure began a core shift as it changed from Windows-based to Linux-based deploy agents for simplicity and cost reasons. Linux build agents quickly became necessary and the primary use of compute for builds on Azure Pipelines supporting Linux-based container development. Many primarily focused Microsoft engineering teams began migrating and using .NET Core on Linux as well to make use of the new container runtime on AWS ECS. This led to the adoption by the internal SPS SRE team of Azure DevOps as an official build and deploy pattern and tool at SPS Commerce.

Through internal conferences and tech meetups, individuals and teams began sharing their experiences, patterns, and success with Azure DevOps + BDP Core as a build and deployment pattern. Team onboarding became more difficult to manage, requiring standardized IAM policy and ticket integration into the process. Security and IAM teams gladly supported it and helped advise on general best practices as we worked to onboard 200+ users across dozens of root projects. The concept that teams could view deployments to all environments from a single view and use centralized infrastructure for it all became very powerful. Teams were enabled to quickly migrate from other tooling, including Jenkins and Drone if they were already using the BDP Core abstraction within their codebases.

Renaming VSTS to Azure DevOps

You may or may not be aware, but “Azure DevOps” was not always its name. In fact, going back in history Azure DevOps first materialized as a TFS (Team Foundation Server) in the Cloud. I had the unfortunate pleasure of building many on-premise TFS instances ranging from 2005, 2008, 2010, and 2012. I remember signing up and using my preview install of TFS in the cloud in 2011, and it was a very exciting day! Eventually, that evolved into Visual Studio Team Services (or VSTS). Visual Studio Team Services was definitely a mouthful and problematic during our rollout of it to the SPS Commerce team. Mainly because “Visual Studio” leaks a Microsoft connotation to it that sounded expensive and probably something specific to “Windows” at the time. Through demos and onboarding of different teams, we were able to shake those concerns quickly.

At some point in the timeline around now, we also begin hearing about Satya Nadella’s internal initiative to move all Microsoft engineering teams to a single engineering platform, that is the entire Azure DevOps tooling suite to standardize their practices under “1ES” (One Engineering System). No doubt an exciting prospect to realize that journey and take its advantages, perhaps something we could achieve someday at SPS Commerce (hint hint).

It was after this, that the rename to “Azure DevOps” put an entirely new challenge in our path for marketing the use of it clearly across our teams at SPS Commerce. At the time we primarily deployed 95% of our infrastructure to AWS Cloud, with a very small footprint in Microsoft Azure Cloud. This left MANY internally thinking that “Azure DevOps” was a tool primarily intended for deployment into the Azure Cloud, and teams thought if they didn’t deploy in Azure, what use did they have for Azure DevOps. That seems like a fair assumption to make based on the name. The reality is of course far different, and Azure DevOps is a very capable platform and product suite that integrates with all clouds, and offers the robust ability to integrate feature sets with many other tools including integrations with JIRA, GitHub, Jenkins, etc. If you desire you can put code in GitHub, build with Jenkins and deploy with Azure Pipelines to AWS without too much effort. Azure DevOps’s capability to be flexible in its integrations no doubt has served it well. Nevertheless, the name was a more significant challenge to deal with than I had expected. Eventually, critical mass was hit in understanding its position and that is less of a problem now (naming is always hard, no matter the industry)!

SPS’ Official CI/CD Platform

Through many other pivotal events, the usage of Azure Pipeline is now not only the officially recommended pipeline technology of choice at SPS, but it is also the only non-deprecated and supported pattern of choice for all build and deploy functionality across multiple platforms and deploy targets. Through the development of a new container runtime backed by AWS EKS, new platform initiatives, pivotal team adoption, and adaptability of usage within the SRE team, it became clear that Azure Pipelines was a good choice for teams to move forward with, and in general, required less maintenance than other self-hosted solutions before it. Make no mistake the SaaS capability is powerful in reducing the load on a team. The hybrid model to be able to use both self-hosted agents (in which containers are provided from Microsoft) alongside cloud-hosted compute provides a good deal of flexibility and capability.

Of course, the cost was a huge consideration in this decision as well. Before Azure Pipelines, Drone was quickly becoming popular given some of its containerized build capability and how that enabled the centralization of builds on shared infrastructure which was decoupled from the dependencies of the codebase. While very advantageous without a release component in earlier versions, this left Drone still as a single piece of the entire CI/CD capability. With Drone’s acquisition by Harness and updated licensing model, it quickly became a realization that further investment in Drone from a cost perspective was not viable. In comparison to Azure Pipelines footprint cost, we continue to pay almost nothing for its direct usage. Through existing Microsoft partnership programs that offer credits and parallel pipelines (one for every enterprise user), self-hosted compute agents, and “Stakeholder” (FREE) licenses that provide full “Azure Pipelines” access, the cost for us to build and deploy across our organization is next to zero. Of course, I left out the fact that our self-hosted agent and compute used for build and deploy is deployed into AWS as EC2 and Containerized agent pools, which have associated costs going towards AWS for that. But nevertheless, this provides incredible capability for an incredibly low cost.

Multi-Stage Declarative YAML Deployments

YAML-based configuration trends are the de facto standard these days. Some days you may just feel like a “YAML engineer”. There is a core reason for this trend. Storing your configuration as code and not clicking around a UI makes your code self-documenting, repeatable, versioned, and a lot less tedious (click/click). In the early days of Azure DevOps, the Build and Release were separate features that both required design and creation through the UI. This was a nice feature for introducing the system. But very quickly, you’ll see it prevents reusability to some degree and can make it difficult to just “see your entire workflow” through Build and Release (though Azure Pipelines tried to do a decent job visualizing that). Nevertheless, moving to declarative YAML-based pipelines was inevitable with the industry trends. Moving the Build functionality was straightforward. But moving release functionality to YAML was a little more difficult. Props to Microsoft to redesigning the entire release workflow to function in a multi-stage declarative YAML Format, but feature parity was and is still not available for the original “Release” feature.

“Builds” was renamed to “Pipelines” and its composition could now be used to orchestrate your entire deployment workflow, from one single YAML file in your repository. The benefits and outcomes of that are desirable for most teams and workflows. It took us some time to transition to that with the same flexibility and capability (at least MVP capability) that was available in Azure Pipeline Releases. While we lost some capability and flexibility, resulting templating patterns and capability better position us for org-level changes, updates, and governance, which is highly desirable for any maturing organization. Most pipelines now function with a single YAML file that builds and deploys all the way through to production using pre-built, reusable, and governed templates and components.

Take a look at a basic build and production deployment:

name: $(version).$(build)
trigger: none
pr: none

resources:
  repositories:
    - repository: templates
      type: github
      name: custom-templates
      endpoint: templates

extends:
  template: base.v1.yml@templates
  parameters:
    bdpFile: .bdp
    stages:
    - stage: COMPILE
      jobs:
      - job: BUILD
        pool: BUILD
        steps:
        - checkout: self
          clean: true
        
        ... BUILD Container, unit tests, etc
          
        - publish: $(Build.ArtifactStagingDirectory)
          displayName: publish deploy artifacts
    
    # deploy to test 
    - template: core/deploy.v1.yml@templates
      parameters:
        environment: test  

    # deploy to prod 
    - template: core/deploy.v1.yml@templates
      parameters:
        environment: prod

Additional templates and custom tasks support the building of containers and language-specific build patterns that can be added in through composition. Metadata and standardization about a service are well known programmatically in the SPS ecosystem that enabling deploying to a new environment is three lines of code to pull in the deploy template and indicate your environment.

This default pattern supports:

Easily deploy to any number of environments (i.e., dev and prod, or dev, test, and prod, etc.)
Identify environments that require manual approval
Automated change management approval workflow integration with JIRA through Azure Pipelines Environment checks
Integrated governance and validation of service metadata against a central registry
Contextual platform variables based on the deployment environment
Integrated semantic versioning model(s)
Integrated pull request context and variables
Enforced security checks for Git branches to be protected, certain names, required templates before deploying to production
Filtered custom task list to remove or identify any improper or insecure usage
Update organizational templates to affect all pipelines

GitHub Actions Comes Along…

Azure Pipelines standardization, capability, and governance components are all working well to remove friction and roadblocks for teams every day who are engineering new services or updates. It was only implied above, but we do not use Azure Repos, but rather GitHub as an organization for version control. Azure Pipelines is then a consumer of our GitHub source. This made for some interesting discussion with regards to the launch of GitHub Actions. GHA of course is a natural evolution of the full suite of capability that GitHub could provide, now a subsidiary within Microsoft. GHA has the power to automate events across the suite of tools that GitHub offers, with one aspect of that being CI/CD for your applications and services. Having the CI/CD capability so close and in a single integrated tool like GitHub is no doubt a strong advantage and strength. As an enterprise GitHub organization, the question must be asked then if a shift and migration towards Azure Pipelines is still the right course of action? To help with that discussion we took the debate to Microsoft and GitHub to inquire on their thoughts further.

Out of the gate, there is some interesting cooperation between Azure Pipelines and GHA. Noticeably is that they look similar from a syntax perspective and in some cases function very similarly too. In fact, as I understand it, early on at least, core agent infrastructure was shared between products. The GHA team and the Azure Pipelines team is in fact a single team focused on delivering functionality to both platforms. I had the opportunity to discuss the roadmap and key features with one of the core architects and program managers for GHA, who also previously was a core contributor to Azure Pipelines Multi-Stage YAML re-envisioning. As you can imagine, he provided some great perspectives. Hands-down, core development was then focused on GHA and less on Azure Pipelines, and the effect on the feature timeline was clear. But what was also clear was the fact that GHA had a LOT of catch-up to do to participate at the enterprise level. Sure, it was new, interesting, and integrated, but it was missing LOT of features we use in Azure Pipelines. At the time it was missing the concepts of “environments” or approvals and is still missing features around reusable templates. With time though, I’m confident features will eventually get to parity or better in some cases. GitHub Actions feature for secure deployments with OpenID Connect is a great example of a powerful new feature only in GHA.

With the information in hand, it was going to be several years before GHA would meet enough feature parity with Azure Pipelines to make it viable for the position and feature capability we had already achieved, not including some core customized features we already had built out or would be coming soon in our own roadmap. Stopping an organization to wait a few years for the potential of a better ecosystem simply did not make sense especially with the value we were already realizing. There is a lot of potential in GHA and look forward to interesting ways we can make use of it, but for the time being and what we know of our roadmap, using GitHub Repos combined with Azure Pipelines really does give us the best of both ecosystems. We take the best-in-class version control and combine it with a powerful mature, enterprise-ready CI/CD platform. This still enables us to take advantage of features like GitHub Advanced Security even outside of the GHA ecosystem. Additionally, our entire enterprise world does not live in GitHub. In fact, we still use and incorporate other systems for Package Management (Azure Artifacts, AWS ECR, JFrog) and Issue Tracking (JIRA) making the advantage of working in a single tool not an immediate realization if we were to switch to GHA as a quick win. Azure Pipelines, as a component of Azure DevOps, continues to grow (with updated feature timelines recently published for 2022) and provides us with better enterprise support with the current state of the tools.

Now & Next

Azure Pipelines at SPS now supports over 50 teams with 400+ engineers, with hundreds of builds and deploys to production daily, across multiple clouds and cloud regions, on 4 different continents. All pivotal infrastructure and services running the SPS Platform and Network are built and deployed through Azure Pipelines. Customized event and auditing are created through custom webhooks and pushed to centralized kinesis stream, pipelines are tracked through association to custom internal service registrations, and change management automation automatically enables production deploys based on the workflow of approvals when required. Pipeline configurations are entirely set up and configured through a single YAML file that enables teams to quickly onboard new pipelines to production fast. As of very recently, platform API documentation is automatically curated and pushed to the centralized SPS API Reference. All this capability and this article add up to an incredible transformation or CI/CD journey at SPS Commerce that is only just beginning.

Moving forward in the cycles ahead, we have a desire to continue to build upon this base and enable enhanced Azure Pipeline agent capabilities and multi-region resiliency, better cost tracking for teams, enhanced API Design First integrated patterns, SSO and automated policy provisioning, and enhancing the flexibility of deployment solutions by reducing proprietary technology where possible.

What Did We Learn?

Sure, we have learned a ton about how to customize and work with advanced orchestration workflows in Azure Pipelines and build out CI/CD patterns. But more specifically, what are some of the core principles or takeaways architecturally you might consider as part of this journey?

Technology Awareness

Having awareness of other technologies within the domain you work in is essential. While at the same time having a system for effective evaluation is important. Without the journey described above, it is highly unlikely that SPS Commerce would have ever considered Azure Pipelines as a tool to solve their CI/CD problems. This identifies how crucial it is to keep up to date with other technologies and patterns in the domain you are solving for. The time may come for another large shift in the technologies you are working with. That may be AWS ECS to AWS EKS or from Drone to Azure Pipelines. The time will come for a shift from Azure Pipelines to GitHub Actions. Keeping that pilot light lit and that evaluation open for emerging technologies while refraining from organizational disruptions until the timing is right is the skill of a good Architect or leader within your organization. Do not pivot to the new and shiny technology without incredible value on the other side. Pivoting requires not just feature parity, but that and then some to make it worth it.

Evolvable Infrastructure & Technology

Azure Pipelines is a great example of a technology that supported our journey at different levels of maturity throughout. Whether through the initial community support, or ease of onboarding through clickable Release UI orchestration creation, or using an organizational reusable template, Azure Pipelines provided us with immense value to take advantage of where we as an organization was at, enabling continued support of legacy implementations and reasonable migration paths to new more valuable patterns. This flexibility and support show the signs of a mature product and tool, which is a huge asset to the enterprise. Even today, while 80% of our pipelines use newer patterns, supporting bespoke older patterns and oddball deployment needs allows us to meet the demands of the other 20% of deployment requirements still within the same ecosystem of tooling.

Evolutionary Architectures

We definitely have some pipeline patterns that did burn us over time. Thinking ahead about how to future-proof your architecture and where some of those demons lie in your infrastructure can have a drastic cascading effect, especially on pipelines and infrastructure in general, as it can be harder to evolve. Looking for areas to apply evolutionary architectural thinking is an incredible help for long-term vision and planning that can really insulate you. The introduction of a single “dependency” that is incorrectly positioned on your build/deploy agents can have long-term effects on your ability to change the platform and move quickly. More concretely containerization and templating are two critical areas that removed some architectural handcuffs for us.

Names Can Be Deceiving

If you learned anything from Microsoft’s choice of naming for VSTS or Azure DevOps, consider the impact the name may have on future marketing or rollout of your product both internally and externally. It can cause a world of pain with no technological bearing.

Developer Experience Matters

With my focus on Developer Experience as of late, it is clear to me more than ever, through this example and many others like it, that the experience of a developer through their day-to-day operations can drastically change the productivity of an organization. Obvious productivity wins and quality of life enhancements can highly motivate even the largest of enterprise change management initiatives at the grassroots level.

-Travis Gosselin, Principal Engineer, Developer Experience

Reference: http://blog.travisgosselin.com/sps-commerce-azure-pipelines…

2022 Technology Internship Positions Now Open!

Wed, 08 Dec 2021 13:50:32 -0500

Our interns are the coolest bunch around, and we are looking forward to meeting the next round of incredible minds for our 2022 program. (Seriously, read about our team from last year and you will see! Goodbye Summer Interns)

TeamSPS is hiring 11 interns for our summer 2022 internship program. You will work as part of our larger technology team of over 300 people around the globe. SPS internships are paid. You will be an active contributing member of the team. You will use the same technologies our engineers use and work side-by-side with them on our production systems running at a massive scale in the cloud. We also believe in growing our interns both professionally and personally. You can expect many learning sessions about how to build your professional network, create an attractive resume, and job searching in today’s market, in addition to improving your technical knowledge. Our intern program is designed for you to continue your learning and gain usable experience. As a result, no one will ask you to get coffee, make copies, or run errands. Instead, you will be expected to ask questions and help us create production-quality code.

There are available positions in Software Engineering, on our Corporate Security, and Technology Learning Services Teams.

Check out this link to learn more about SPS Internships!

Plan, Plan & Prepare for the Unexpected: Cyber Week 2021

Fri, 26 Nov 2021 13:16:57 -0500

My husband, Joe, wanted to buy a tractor for over five years to use at our cabin to grade the mile-long road we maintain and to do other important and cool stuff that he assures me can only be done with a tractor. For over five years, he has researched, planned, and hunted for that perfect used tractor with low hours of use, at a price he liked. My only requirement was it had to be a John Deere…just because. So, he planned, and planned, and planned. For over five years.

Finally, this month, he found the perfect used John Deere 2032R compact tractor and scheduled to pick it up from the seller. He was so excited and grinning like a little kid. But, on the night before he was to pick it up, he hit a major snag. He couldn’t swap out his truck’s trailer hitch because it was rusted in, and hours of pounding on it didn’t move it. Even with years of planning, unexpected incidents can happen and throw you off. Luckily, we were able to pivot and have the tractor delivered to my parents’ house, which was just 10 minutes away from the seller, but still over 2 hours from the cabin. Joe had to bring the truck to a welder to get the hitch removed. We held a very quick retrospective on this incident and decided that the hitch will be removed after every use from now on. 😀 No incident report will be needed for this, and the tractor has since made its way to the cabin.

Here’s where I’m going to tell you that all of my husband’s tractor planning and last-minute trailer hitch incident is a bit like preparing for and managing Cyber Week. Just go with me on this one.

SPS spends months, and even years, preparing for Cyber Week and the Peak Season that surrounds that one week of the year. As the world’s retail network, we are keenly aware of the importance of this time of year for the retail industry, our customers, and SPS. We put extra focus and care into planning for scale and resiliency and preparing for the unexpected – all to ensure our part in the supply chain just works.

As soon as we finished Cyber Week 2020, we held a retrospective and began planning for this year’s Cyber Week 2021 and beyond. We tracked critical system metrics, collaborated with Sales and Customer Success teams, estimated peak volumes during our busiest hours and days, and performance tested to beyond our highest estimates. We do some serious planning and preparation.

But we also prepare our processes and our people to handle the unexpected. Yes, just like that trailer hitch being rusted in, things can go wrong during Cyber Week. That’s why we have a robust Incident Management process, complete with defined roles, responsibilities, and communication and escalation methods. We augment that strong process by training our staff and providing coaching for people in new roles, such as Incident Manager and Customer Captain.

When Cyber Week arrives, we add an increased level of focus for our teams by implementing a process called Hypercare, which means hyper-focus on system performance and hyper visibility to metrics. Hypercare decreases the time to identify and respond to potential system issues while improving the efficiency of our communication channels.

Through all of this, we have fun being part of #TeamSPS. We have a virtual Hypercare Lounge where people around the globe hang out to watch our production systems, discuss application metrics, and decompress by sharing fun virtual backgrounds. We live our company values even at times of stress. We realize that Results Matter and that we Succeed Together. We can also be Obsessed With Customers and value Employees Come First at the same time.

We are ready for our customers! I might not be grinning like a little kid just yet (that’ll be after Cyber Week), but I’m excited. Happy holiday shopping!

– Amy Patton, VP Technology

SPS talks about Feature Flagging in a customer panel for LaunchDarkly

Tue, 23 Nov 2021 14:29:00 -0500

Recently, I was invited to join a panel of customers for an event called “Launch and Learn.” It was a dual-purpose event that included marketing for LaunchDarkly and also questions from their engineers about how customers use their products. SPS is a long-time LaunchDarkly customer and jumped at the opportunity to help them out in this way.

If you want to hear more about how SPS uses LaunchDarkly to flag our new features and continually release code, here’s the video of that event:

Peace of Mind & Two SPS Engineers Were the Big Winners at Hack The Gap:

Wed, 03 Nov 2021 14:56:15 -0500

Two of our Software Engineers from our Minneapolis office, Ana Knickerbocker (find her on Twitter too!) & Jenna Truong took part in a hackathon in October through Hack the Gap. We asked them to share a little about the hackathon and what they did. They were on a team of 4 individuals and their project ended up winning, aligning nicely with the SPS Value of Win Today, Win Tomorrow. 🎉🏆

About Hack the Gap:

“At Hack the Gap our focus is increasing equity in technology by highlighting diverse voices and perspectives. This includes women, non-binary, black, indigenous, and people of color. We do this through hackathons, programming, and investing in technology built by and for women, non-binary and BIPoC peoples.” (https://www.hackthegap.com/)

About Hack the Gap Hackathon:

Hack the Gap hosts hackathons for women and non-binary individuals. These are usually different from a typical hackathon as they might occur in the middle of the week or over a longer period of time to better accommodate the schedules of mothers, women, and non-binary individuals that they cater to.

Our Project:

Our project was built to address a gap in available task messaging apps. In our demo we highlighted several key points:

Are you highly skilled in the art of forgetfulness? I know I am! We’re here to offer you some Peace of Mind.
Peace of Mind is a task management app that won’t let you forget. You can get an indefinite number of alerts via text messages, emails, and phone calls so you can’t ignore the task at hand.
The next iteration will offer more relentless notifications. For example, “text me every 1 minute until the task is completed” or “text me every 5 minutes for the first hour minutes, then call me every minute until the task is completed.”

We chose this project because time management has gotten tricky since the start of the pandemic. People are experiencing difficulty perceiving the passage of time, and often feeling “brain fog.” Many tasks are too important to be delayed or forgotten, like taking medications, caring for others, or caring for yourself. We believe this application will help some stop worrying about forgetting, and bring some peace of mind.

The project was initially generated using Nx. We used a Node/Express/TypeScript backend with a React/TypeScript frontend. On the frontend we incorporated components from Bootstrap, Twillio & Ant Design. One of our teammates was a designer who also created some beautiful logos! The scheduling of alerts was accomplished by calculating the next alert DateTime, saving that DateTime to the database, and running a query against the database every minute using a cron job. The alerts were sent out via Twilio/SendGrid APIs.

What we learned:

On the frontend, in addition to building a UI from scratch for the first time, we learned about incorporating new design systems. Ana created 2 databases (Postgres > RethinkDB > Postgres) and deployed both of them! We also learned that no matter how good the last demo recording was, you’re likely to just keep re-recording up until the deadline. (Two more SPS Values: Thirst For Growth and Know More To Be More.)

Since we wanted our tool to function as an app instead of just a webpage, in the future we hope to learn how to make this a better cross-platform tool. For example, we would like to include recurring tasks, symptom and mood tracking, and a caregiver mode where you can set alerts for others. Additionally, the API will be made public for easy integration with applications like Trello, Google Calendar, Siri, and Alexa.

What we enjoyed:

The highlight of the hackathon: our team won! (Again, demonstrating those SPS Values: Succeed Together & Win Today, Win Tomorrow.) We also enjoyed the opportunity to work with new teammates located across the country (and world) in Michigan and Spain! A beginner developer on our team had never seen an application built from scratch before! It was fun showing her how everything worked, helping her contribute, and answering all her excellent questions. It was also refreshing to work on a side project where we could be as creative and as innovative as we wanted.