Lokesh Nanda

29 June 2025 · graph-databases · dbt · aws · llm · books

Graph databases, dbt hands-on, AWS Summit — and starting this notes site

24 June 2025

Came across a graph database — DGraph. Found it to be a very interesting open-source project, so I explored its competitor, Neo4j Aura:

  • Created an account in Neo4j Aura online. Self-hosting was also an option, but went ahead with this.
  • Created an integration to extract wiki pages using LangChain and extract entities:
from dotenv import load_dotenv
import os
from langchain_neo4j import Neo4jGraph
from langchain_community.document_loaders import WikipediaLoader
from langchain.text_splitter import TokenTextSplitter
from langchain_openai import ChatOpenAI
from langchain_experimental.graph_transformers import LLMGraphTransformer

load_dotenv()

NEO4J_URI = os.environ["NEO4J_URI"]
NEO4J_USERNAME = os.environ["NEO4J_USERNAME"]
NEO4J_PASSWORD = os.environ["NEO4J_PASSWORD"]
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

chat = ChatOpenAI(api_key=OPENAI_API_KEY, temperature=0, model="gpt-4o-mini")

kg = Neo4jGraph(url=NEO4J_URI, username=NEO4J_USERNAME, password=NEO4J_PASSWORD)

# read the wikipedia pages
raw_documents = WikipediaLoader(query="The Indian Politics").load()

# define chunking strategy
text_splitter = TokenTextSplitter(chunk_size=512, chunk_overlap=24)
documents = text_splitter.split_documents(raw_documents[:3])

llm_transformer = LLMGraphTransformer(llm=chat)
graph_documents = llm_transformer.convert_to_graph_documents(documents)

# store to neo4j
res = kg.add_graph_documents(
    graph_documents,
    include_source=True,
    baseEntityLabel=True,
)
  • Doing the same in DGraph would require writing a wrapper, as it doesn’t have out-of-the-box integration like Neo4j does.

25 June 2025

dbt hands-on. Wanted to explore and do hands-on in dbt, following a data engineering project with Snowflake:

  • dbt doesn’t extract or load data — it focuses on the transformation step in ETL, and runs on top of a data warehouse (compute happens in the DW).
  • I still wondered: I can keep .sql in my source repo and control it there, so why dbt? The benefits: documentation out of the box, reusability, a built-in testing framework, dependency management, modularity.
  • dbt Cloud vs dbt Core: Core doesn’t include the built-in semantic layer. It can be done via MetricFlow, but exposing it via API is a Cloud feature. Cube.dev is an alternative if you want to stick to dbt Core for the semantic layer.
  • Models — SQL scripts that can reference other models; run with dbt run. Materializations per folder: table, view, incremental ({% if is_incremental() %} — you write the logic), ephemeral (in-memory only, but still referenceable).
  • Seed — not really for large data; fine for small datasets. Sources — helpful for documentation (sources.yml under models/).
  • Snapshots — built-in type-2 SCD implementation. Very useful if you need SCD in the warehouse.
  • Tests — generic (defined in models/schema.yml) and singular (custom SQL under tests/); severity is configurable (warn/error).
  • Docs — dbt docs generate && dbt docs serve. Loved it; the lineage view is very good.
  • Macros (reusable functions), hooks (logging, permissions around jobs) and custom operations (e.g. add a partition at a specific time).
  • CI/CD should be set up in the repo; Airflow can trigger models on schedule. Model organisation and naming conventions are critical — inspiration here.

Publishing these notes. Learnt about GitHub Pages and hosting static websites on it — and created the first version of this weekly-notes site. Also identified that the easiest and most economical way to get a domain is Cloudflare.

Reading: The Snowball. Some concepts worth noting:

  • The “Inner Scorecard” vs “Outer Scorecard”: live by your own values, not society’s expectations.
  • Ben Graham — need to read The Intelligent Investor again.
  • Value investing: buying cheaper than the intrinsic value.

Tools spotted: Cube.dev’s dbt recipe as a dbt-core semantic layer, and Flowershow for hosting markdown directly as a website from GitHub.

26 June 2025

Notes from AWS Summit India 2025.

Gen AI:

  • Where does Amazon actually use gen AI in production internally? Chatbots; audio, video & image generation tools; inventory management.
  • Amazon Q Developer — very similar to Cursor and Copilot. Amazon Q Business — out-of-the-box chatbot on top of your internal data; integrates with SharePoint. We did something similar with Microsoft Copilot but found indexing costs really high — need to check how different this is.
  • Amazon builds its own AI chips for cost reduction: Trainium2 and Inferentia2.
  • Bedrock Agents — simple multi-agent flows without code, pointing at a knowledge base; tools connect via Lambda functions.
  • SageMaker HyperPod — LLM fine-tuning with hardware support; 6-month reserved capacity showed ~68% cost reduction. Uses Slurm for cluster management. Perplexity AI and Stability AI train on it.

Data:

  • Amazon is going the Azure-Fabric way with SageMaker Unified Studio (all-in-one platform) — a rebrand, Iceberg-native but supporting Hudi, Delta and Iceberg.
  • Zero-ETL integration and federated querying; zero-ETL for Salesforce via Glue with automatic incremental setup — no ingestion framework needed.
  • Valkey (the community Redis replacement since the 7.2 licence change) on ElastiCache (microsecond writes, serverless option) and MemoryDB (in-memory with durability via multi-AZ transaction log).
  • Bedrock can be called directly from a Redshift SQL query — e.g. create a sentiment score with just SQL. Good use case.
  • S3 Table buckets — namespaces and tables with automatic compaction, snapshots and file cleaning; good Iceberg support with Kinesis/Firehose ingestion, query via Athena.
  • QuickSight has good Amazon Q integration for out-of-the-box dashboard creation.

LLM field notes:

  • An interesting use case in the security space — multi-model threat investigation.
  • Insurance claim processing is an interesting use case for gen AI apps.
  • I observed that if one agent has too many tasks and tools, the codebase gets hard to maintain and it starts hallucinating. Multi-agent architecture helps.

27 June 2025

  • Model distillation: a smaller, more efficient model (student) learns to mimic a larger, more powerful model (teacher). Main benefit: reduced cost.