Generalist Rater

This is a remote position.

Role: Generalist Rater

Remote: USA

Contract

Responsibilities:

● Design and write effective prompts that elicit desired responses from LLMs, considering

different tasks, contexts, and target audiences.

●Generate creative and informative text formats, including stories, articles, summaries,

dialogues based on guidelines.

●Evaluate and compare the quality of LLM-generated responses based on criteria such

as fluency, coherence, factuality, creativity, and bias.

●Analyze LLM outputs to identify strengths and weaknesses, and provide feedback to

improve model performance.

●Collaborate with engineers, researchers, and product managers to refine prompting

strategies and evaluation metrics.

●Contribute to the development of best practices and guidelines for prompt engineering

and LLM evaluation.

Requirements

Qualifications:

●Bachelor's degree in English, Creative Writing, Linguistics, Computer Science, or a

related field.

●3+ years of experience in content writing, copywriting, or a related field.

●Strong understanding of LLM capabilities and limitations.

●[Preferred]: experience with video generation tasks

●Excellent writing and editing skills, with the ability to adapt to different styles and tones.

●Experience with prompt engineering and evaluating LLM outputs is a plus.

●Understanding of statistics and data analysis is a plus.

●Familiarity with data annotation and labeling tools is a plus.

●Strong analytical and problem-solving skills.

●Ability to work independently and as part of a team.

●Can-do attitude, willingness to learn, and growth mindset.

Benefits

Preferred Qualifications:

●Experience working with different LLM architectures and frameworks.

●Knowledge of NLP concepts and techniques.

●Familiarity with human evaluation methods for LLMs.

●Experience in a research or academic setting.

●Tech savvy: ability to read json not required, but we need candidates willing to grow their

technical abilities

●Bilingual preferred (Chinese, Spanish, Portuguese, Arabic, French, German).