Definition
- LLM (Large Language Model) is a neural network with billions of parameters, trained on text from books, code, and the web, that generates text by predicting the next token.
- Token is a piece of text (roughly a word or subword) that the model processes; both input and output are measured in tokens.
- Prompt is the input text you send to the LLM, including instructions, context, and examples.
- Prompt engineering is the practice of crafting effective prompts to get accurate, relevant outputs.
- Hallucination is when the LLM generates plausible but fincorrect information, including non-existent API methods or wrong library versions.
Think of an LLM as the most well-read consultant at the National Bank of India. They have read every Java textbook, every Spring Boot guide, every StackOverflow answer, and every banking regulation document. When you ask them a question, they synthesize what they have read and give you a well-structured answer. But they can also confidently make things up: they might reference a method that does not exist or a library that was never released. You always verify their advice before using it in production.
Understanding the Concepts
What is an LLM and how does it work?
An LLM is a neural network (typically a Transformer architecture) trained on massive text datasets. It learns patterns in language and code. When you give it a prompt, it generates output by predicting the most probable next token, one token at a time.
Key characteristics:
- Training data. Billions of documents: books, code repositories, documentation, web pages.
- Parameters. Billions of learned weights that encode patterns. GPT-4 has estimated hundreds of billions of parameters.
- Context window. The maximum number of tokens the model can process at once. Ranges from 8K to 200K tokens depending on the model.
- Temperature. Controls randomness. 0.0 = deterministic, 1.0 = creative. For code generation, use 0.0-0.2.
package com.bank.ai;
public class LLMBasicsDemo {
public static void main(String[] args) {
System.out.println("How an LLM generates code:");
System.out.println();
System.out.println("Input prompt:");
System.out.println(" 'Write a Java method that validates an IFSC code'");
System.out.println();
System.out.println("Model processes:");
System.out.println(" Token 1: 'public' (high probability after 'Java method')");
System.out.println(" Token 2: 'static' (common for utility methods)");
System.out.println(" Token 3: 'boolean' (validates -> returns true/false)");
System.out.println(" Token 4: 'isValid' (naming convention)");
System.out.println(" ... continues token by token ...");
System.out.println();
System.out.println("Output:");
System.out.println(" A complete method with regex validation and Javadoc");
}
}
How an LLM generates code:
Input prompt:
'Write a Java method that validates an IFSC code'
Model processes:
Token 1: 'public' (high probability after 'Java method')
Token 2: 'static' (common for utility methods)
Token 3: 'boolean' (validates -> returns true/false)
Token 4: 'isValid' (naming convention)
... continues token by token ...
Output:
A complete method with regex validation and Javadoc
How do Java engineers use LLMs for code generation?
LLMs generate boilerplate code, implement algorithms, and scaffold entire classes. The key is providing specific context in the prompt.
package com.bank.validation;
import java.util.regex.Pattern;
/**
* Generated with LLM assistance, reviewed and tested by the team.
* IFSC (Indian Financial System Code) format: 4 letters + 0 + 6 alphanumeric.
* Example: SBIN0001234 (State Bank of India, branch 001234)
*/
public class IFSCValidator {
private static final Pattern IFSC_PATTERN =
Pattern.compile("^[A-Z]{4}0[A-Z0-9]{6}$");
public static boolean isValid(String ifsc) {
if (ifsc == null || ifsc.isBlank()) {
return false;
}
return IFSC_PATTERN.matcher(ifsc.toUpperCase()).matches();
}
public static void main(String[] args) {
System.out.println("SBIN0001234: " + isValid("SBIN0001234")); // true
System.out.println("HDFC0002345: " + isValid("HDFC0002345")); // true
System.out.println("INVALID: " + isValid("INVALID")); // false
System.out.println("SBI00012: " + isValid("SBI00012")); // false (too short)
System.out.println("null: " + isValid(null)); // false
}
}
SBIN0001234: true
HDFC0002345: true
INVALID: false
SBI00012: false
null: false
How do you integrate an LLM into a Java backend?
You call LLM APIs over HTTP. Most providers (OpenAI, Anthropic, AWS Bedrock) expose REST endpoints that accept a prompt and return a completion.
package com.bank.ai;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
public class LLMIntegrationDemo {
private static final String API_URL = "https://api.openai.com/v1/chat/completions";
public static void main(String[] args) throws Exception {
String apiKey = System.getenv("OPENAI_API_KEY");
String requestBody = """
{
"model": "gpt-4o",
"messages": [
{"role": "system", "content": "You are a banking domain expert."},
{"role": "user", "content": "Explain the NEFT transfer process in 3 sentences."}
],
"temperature": 0.2,
"max_tokens": 200
}
""";
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(API_URL))
.header("Authorization", "Bearer " + apiKey)
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(requestBody))
.build();
HttpClient client = HttpClient.newHttpClient();
HttpResponse<String> response = client.send(
request, HttpResponse.BodyHandlers.ofString());
System.out.println("Status: " + response.statusCode());
System.out.println("Response: " + response.body());
}
}
Status: 200
Response: {"choices":[{"message":{"content":"NEFT (National Electronic Funds Transfer) processes transactions in hourly batches..."}}]}
What are the risks and limitations of using LLMs?
| Risk | Description | Mitigation |
|---|---|---|
| Hallucination | Generates plausible but incorrect code or facts | Always review and test generated code |
| Security | May generate code with SQL injection or hardcoded secrets | Run security linting on all generated code |
| Outdated knowledge | Training cutoff means it may not know the latest Java features | Provide current docs in the prompt (RAG) |
| Data leakage | Sending proprietary banking code to an external API | Use private deployments or on-premise models |
| Non-determinism | Same prompt may give different outputs | Set temperature=0 for code generation |
package com.bank.ai;
public class LLMRisksDemo {
public static void main(String[] args) {
System.out.println("LLM hallucination example:");
System.out.println();
System.out.println("Prompt: 'How do you use BigDecimal.round() in Java?'");
System.out.println();
System.out.println("LLM output (WRONG):");
System.out.println(" BigDecimal result = amount.round(2, RoundingMode.HALF_UP);");
System.out.println();
System.out.println("Actual Java API (CORRECT):");
System.out.println(" BigDecimal result = amount.setScale(2, RoundingMode.HALF_UP);");
System.out.println();
System.out.println("The LLM confidently used a method signature that does not exist.");
System.out.println("Always compile and test generated code.");
}
}
LLM hallucination example:
Prompt: 'How do you use BigDecimal.round() in Java?'
LLM output (WRONG):
BigDecimal result = amount.round(2, RoundingMode.HALF_UP);
Actual Java API (CORRECT):
BigDecimal result = amount.setScale(2, RoundingMode.HALF_UP);
The LLM confidently used a method signature that does not exist.
Always compile and test generated code.
What are common mistakes when using LLMs?
Mistake 1: Trusting generated code without review
Wrong: copying generated code directly into production.
Correct: treat LLM output as a draft. Review for correctness, security, and adherence to your coding standards. Run tests.
Mistake 2: Sending sensitive data in prompts
Wrong: including real customer data, API keys, or production credentials in LLM prompts.
Correct: use anonymized or synthetic data. Use private model deployments for sensitive banking workloads.
What should a senior engineer know?
- Token economics. GPT-4o charges per input and output token. A 1000-token prompt with a 500-token response costs roughly $0.01-0.03. For high-volume banking automation, cost optimization matters: shorter prompts, cached responses, and smaller models for simple tasks.
- Fine-tuning vs RAG. Fine-tuning trains the model on your specific data (expensive, static). RAG retrieves relevant documents at query time and includes them in the prompt (cheaper, always current). For banking, RAG with internal documentation is usually the better choice.
- Evaluation and testing. Build automated evaluation suites for LLM outputs. For code generation, compile and run generated code against test cases. For text generation, use LLM-as-judge or human evaluation.
- Guardrails. Implement input and output filters to prevent prompt injection, detect sensitive data leakage, and validate that generated code follows security policies.
- Model selection. Use the smallest model that meets quality requirements. GPT-4o for complex reasoning, GPT-4o-mini or Claude Haiku for simple tasks like formatting or classification. This reduces cost and latency.
How do you evaluate LLM output quality for banking use cases?
LLM output must be verified before use in banking contexts. You need evaluation datasets with known-correct answers and automated checks for factual accuracy, code correctness, and security.
package com.bank.ai;
public class LlmEvaluation {
public static void main(String[] args) {
System.out.println("LLM evaluation checklist for banking:");
System.out.println(" 1. Factual correctness against known answers");
System.out.println(" 2. Code compiles and passes tests");
System.out.println(" 3. No sensitive data in output (account numbers, passwords)");
System.out.println(" 4. SQL injection or code injection absent");
System.out.println(" 5. Consistent with banking regulations");
System.out.println(" 6. Latency within acceptable budget");
}
}
Expected output:
LLM evaluation checklist for banking:
1. Factual correctness against known answers
2. Code compiles and passes tests
3. No sensitive data in output (account numbers, passwords)
4. SQL injection or code injection absent
5. Consistent with banking regulations
6. Latency within acceptable budget
What are common interview follow-up questions?
- What is the difference between fine-tuning and prompt engineering?
- How do you prevent prompt injection attacks?
- What is a token and how does context window size affect LLM usage?
- How do you evaluate the quality of LLM-generated code?
- What are the compliance considerations for using LLMs in banking?
Visual Overview
LLM Request Flow
==================
Java Application LLM API (OpenAI/Bedrock)
──────────────── ────────────────────────
│ Build prompt: │
│ system + user message │
│ │
│ POST /chat/completions │
│ {model, messages, ──> │ Tokenize input
│ temperature: 0.2} │ Process through layers
│ │ Generate tokens
│ <── │ {content: "...", usage: {tokens: 500}}
│ │
│ Parse response │
│ Validate output │
│ Use in application │
LLM Use Cases for Banking Engineers
=====================================
┌──────────────────┐
│ Code Generation │ Write boilerplate, APIs, tests
├──────────────────┤
│ Debugging │ Explain errors, suggest fixes
├──────────────────┤
│ Documentation │ Generate Javadoc, README, API docs
├──────────────────┤
│ Code Review │ Spot bugs, suggest improvements
├──────────────────┤
│ Legacy Code │ Explain old code, plan migration
└──────────────────┘
Key Takeaway
Large language models are a new tool in the Java backend engineer's toolkit. They generate boilerplate code, write tests, explain legacy systems, and create documentation. At the National Bank of India, engineers use LLMs to scaffold Spring Boot services, write unit tests for transfer logic, and generate Javadoc for existing code.
The critical rule is verification. LLMs hallucinate. They generate code that looks correct, compiles, and even passes a casual review, but contains subtle logic errors or uses methods that do not exist. Every piece of generated code must be reviewed, compiled, and tested before it enters the codebase.
Integrating LLMs into a Java backend is straightforward: call a REST API with a JSON payload containing the prompt, model selection, and parameters. The response contains the generated text and token usage. For banking applications, use private model deployments or providers like AWS Bedrock that keep data within your cloud account.
Prompt engineering is the skill that makes LLMs useful. A vague prompt gives vague results. A prompt that includes the class name, the method signature, the business rule, the domain (banking), and the expected output format gives production-quality results. Include examples in the prompt for the best results.
One sentence to remember: LLMs generate useful code drafts, but always verify, test, and review before using them in a banking application.
---