Log Analysis & Git Blame

Slide 1: Log Analysis & Git Blame

On-screen

Log Analysis & Git Blame

Finding signals without pointing fingers

Narration

Anna: Logs are where an incident timeline actually comes from, and git history is where a code change gets explained.
Greg: Both are also easy to misuse. One buries you in noise, and the other has the word "blame" in the command name.
Anna: Which is an unfortunate piece of naming, because the tool is genuinely useful and the instinct it invites is not.
Greg: So we'll cover finding signal in logs, and using history to understand a decision rather than to identify a culprit.

Slide 2: Why analyze logs?

On-screen

Why analyze logs?

Logs capture the raw truth of how software behaves—"where your application goes to confess its sins." DevOps engineers, SREs and L2/L3 support rely on them daily. Each entry has a timestamp and level like DEBUG, INFO, WARN or ERROR. By lining these up you can spot patterns that metrics alone miss. Imagine debugging a checkout process that fails randomly. You might see 02:25 INFO user initiated payment, 02:30 WARN database lock on orders table and finally 02:31 ERROR payment service timeout after 30s. The timeline reveals that the lock triggered the timeout. Logs validate hunches, reveal memory spikes before incidents and provide evidence for root cause analysis. Because logs may contain personal data, follow your organisation's retention and privacy rules when storing or sharing them.

Narration

Anna: Logs are the storybook of every application. Each entry marks what happened and at what severity level.
Greg: Right, without them we'd be guessing whether a failure came from a bad deployment or a hardware hiccup.
Anna: Remember the checkout bug we chased for days? The logs finally showed payment timeouts minutes after a database lock warning.
Greg: Once we lined those timestamps up, the cause was obvious, and we avoided hours of finger-pointing.
Anna: That's why we dig through logs even when it's tedious. They turn hunches into hard evidence and help us spot patterns early.
Greg: We've all stared at a wall of red ERROR messages at 2 AM wondering where to even begin.

Slide 3: Log analysis techniques

On-screen

Log analysis techniques

Start small with grep "ERROR.*timeout" app.log | tail -20 to grab the last failed requests. As your stack grows, aggregators like the Elastic Stack or Splunk correlate events across microservices. Free tools like ELK work well for hobby projects while paid options offer enterprise support. Structured JSON logs make queries reliable; unstructured text demands regex tricks. Many teams tag each entry with request IDs so they can trace a user's path through distributed services. Distributed tracing passes these IDs across services. Dashboards show spikes in WARN or ERROR levels and whether a memory surge in Service A triggered cascading failures in Services B–F. Aggregators collect logs via lightweight agents into central storage, enforce retention rules and integrate with ServiceNow or Jira for incident tickets. Keep DEBUG logs for only a short time, since excessive logging can slow applications and waste storage.

Narration

Anna: When the app is small, a quick grep for "ERROR" usually does the trick.
Greg: But how do you even know where to start looking in a 10GB log file?
Greg: Once you have several services, tools like Elastic or Splunk become essential.
Anna: They let you search structured fields and follow one request across many logs.
Greg: We tag every entry with a request ID so we can trace a user's journey end to end.
Anna: Dashboards help spot patterns too. A jump in WARN logs might reveal memory pressure before anything crashes.
Greg: Whatever tool you use, keep enough history so you can go back and learn from incidents.

Slide 4: Git blame: friend or foe?

On-screen

Git blame: friend or foe?

Think of blame as the ultimate "it wasn't me" detector. git blame shows who last modified each line of code. It's powerful context for understanding design choices but can feel accusatory if used carelessly. People often worry that blame will be used to point fingers instead of improving the code. The reality is that the last editor may have been fixing someone else's bug or operating under tight deadlines. Treat blame information like fingerprints at a crime scene—useful evidence but not the full story. Combine it with commit messages, issue history and a conversation with the author before drawing conclusions. And sometimes it's best not to run blame at all when emotions are high or the goal is simply to refactor.

Narration

Anna: git blame shows who last touched a line, but that alone doesn't prove responsibility.
Greg: The author may have been fixing someone else's bug or working with incomplete specs.
Anna: When we spot a risky change, we message the contributor and ask what problem they were solving.
Anna: Remember when we blamed the database for three hours before realizing it was a typo in the config?
Greg: Usually we uncover useful context, like a last-minute hotfix that forced a quick decision.
Anna: We also use options like -w to ignore whitespace or -C to track code moved between files.
Greg: Used kindly, blame provides insight without turning conversations into witch hunts.

Slide 5: Using Git blame constructively

On-screen

Using Git blame constructively

When you do use blame, pair it with open dialogue. Suppose a function crashes on null inputs and you run git blame src/user-service.py -L 45,55 -w. Reach out to the contributor and ask what constraints they faced when writing that code. Maybe they were patching a production bug or following outdated requirements. Capture the lessons learned in commit messages or design docs so others won't repeat the mistake. Remember to respect privacy; if a blame trail uncovers systemic issues like unrealistic deadlines or missing reviews, escalate through your team's retrospective process rather than confronting individuals in public channels.

Narration

Anna: When you do reach for blame, pair it with conversation. Say a function crashes on null inputs and the history points at one commit.
Greg: Go and ask that contributor what constraints they were working under. Often they were patching a production bug at speed, or following requirements that have since changed.
Anna: Then capture what you learn in the commit message or a design doc, so the next person meets the reasoning instead of repeating the mistake.
Greg: And if the trail exposes something systemic, unrealistic deadlines or missing reviews, take it to the retrospective. Not to a public channel, and not to the individual.

Slide 6: Example workflow

On-screen

Example workflow

Begin by reviewing logs to trace a symptom back to the code. Imagine a spike of ERROR: database connection timeout entries. After confirming that network latency wasn't the culprit, you check the commit history and see a recent change to connection pooling. Running git blame on that file reveals who altered the pool size. You chat with them to understand the reasoning and learn that the change fixed another outage months ago. Together you run new tests, adjust the pool size and update documentation. Finally, record the incident in your ticketing system and ensure the logs are stored according to retention policy so future teams can learn from them.

Narration

Anna: Here's a typical investigation. We start by scanning the logs for errors like "database connection timeout".
Anna: It can feel stressful when production is down, so having a checklist keeps everyone calm.
Greg: If network metrics look normal, we examine recent commits that touched the connection pool.
Anna: Running git blame on that section shows who adjusted the pool size.
Greg: Instead of blaming them, we ask what issue they were trying to solve and if it's still relevant.
Anna: Together we test new settings, update the documentation, and note everything in the ticket.
Greg: Saving those logs and discussions means the next team understands why we made each change.

Slide 7: Key takeaway

On-screen

Key takeaway

Logs show what happened and when, while Git blame hints at why. Combine them to build a narrative that explains both the technical symptoms and the human decisions behind them. Use tools like Splunk or the "-C" flag in git blame responsibly: keep sensitive data private and open a dialogue before drawing conclusions. Over time these practices shorten incident response, reveal recurring issues, and demonstrate progress with metrics like MTTR. Most importantly, they encourage a culture where admitting mistakes leads to stronger systems rather than shame. Treat these tools as instruments for learning and growth, not as weapons for calling people out.

Narration

Anna: Logs show what happened, and blame hints at who changed the code and why.
Greg: Used together, they help us resolve issues quickly without turning the post‑mortem into a witch hunt.
Anna: We also respect privacy, follow retention rules and document lessons learned.
Greg: That builds trust. People feel safe admitting mistakes, so the whole team improves.
Anna: The goal isn't to catch someone out; it's to make the system stronger after each incident.
Greg: Treat logs and blame as tools for insight, not weapons, and they'll guide your career as much as your code.