What is new Deepseek version 3?

DeepSeek, a Chinese AI startup, has been making significant strides in the artificial intelligence landscape, and its latest offering, New Deepseek version 3 features, is generating considerable buzz. This new model is not just an incremental upgrade; it represents a leap forward in open-source AI, challenging established industry giants and pushing the boundaries of what's possible with large language models (LLMs). This article will delve into the key aspects of DeepSeek V3, exploring its architecture, performance, and potential impact.

DeepSeek V3: A New Benchmark in Open-Source AI

DeepSeek V3 is an ultra-large open-source AI model boasting 671 billion parameters. This massive scale allows it to handle complex tasks and achieve exceptional performance. Released under the company's license via Hugging Face, the model has demonstrated its ability to outperform leading open models such as Meta's Llama-3.1 and Alibaba's Qwen, while also rivaling closed-source models like OpenAI's GPT-4o and Anthropic's Claude 3.5. This achievement underscores the growing parity between open and closed-source AI models, fostering competition and reducing reliance on monopolistic players.

Credit: img.i-scmp.com

Advanced Architecture and Efficiency

One of the key new Deepseek version 3 features is its use of a mixture-of-experts (MoE) architecture. This allows the model to activate only 37 billion of its 671 billion parameters for any given task, enabling robust performance while maintaining efficiency in both training and inference. This design ensures that the model doesn't require excessive computational power for every task. Further innovations include an auxiliary loss-free load-balancing strategy, which ensures balanced use of the model’s neural networks, and multi-token prediction (MTP), which triples generation speed to 60 tokens per second. This makes DeepSeek V3 not only powerful but also remarkably fast.

Training and Cost-Effectiveness

DeepSeek V3 was trained on 14.8 trillion diverse tokens, extending context lengths to 128,000 tokens in a two-stage process. Post-training refinements included supervised fine-tuning and reinforcement learning, aligning the model with human preferences while preserving a balance between accuracy and generation length. The model's development cost totaled $5.57 million, leveraging multiple optimizations such as the FP8 mixed precision training framework and DualPipe parallelism. In contrast, similar projects like Llama-3.1 required over $500 million, highlighting DeepSeek-V3’s cost-efficiency. This cost-effectiveness is a significant advantage, making advanced AI more accessible to a wider range of users and organizations.

Benchmarking Excellence

DeepSeek’s benchmarks reveal DeepSeek-V3 as a strong open-source AI model currently available. It outperformed open counterparts Llama-3.1-405B and Qwen 2.5-72B and rivaled closed-source models like GPT-4o on most tasks. Notably, its performance on Chinese and math-focused benchmarks was unmatched, scoring 90.2 on Math-500, with Qwen trailing at 80. This demonstrates a clear strength in specific areas, making it a versatile tool for various applications. However, it's important to note that Anthropic’s Claude 3.5 maintained an edge in specific tasks like MMLU-Pro and SWE Verified, leaving room for future advancements in open-source AI.

DeepSeek Coder: A Powerful Coding Companion

DeepSeek also offers DeepSeek Coder, a series of code language models trained from scratch on 2T tokens, with 87% code and 13% natural language in both English and Chinese. This model comes in various sizes ranging from 1B to 33B versions. Each model is pre-trained on a project-level code corpus using a window size of 16K and an extra fill-in-the-blank task, which supports project-level code completion and infilling. This combination of large-scale training and specialized techniques allows DeepSeek Coder to achieve state-of-the-art performance among open-source code models on multiple programming languages and various benchmarks.

Key Features of DeepSeek Coder

• Massive Training Data: Trained from scratch on 2T tokens, including 87% code and 13% linguistic data in both English and Chinese.
• Highly Flexible & Scalable: Offered in model sizes of 1B, 5.7B, 6.7B and 33B, enabling users to choose the setup most suitable for their requirements.
• Superior Model Performance: State-of-the-art performance among publicly available code models on HumanEval, MultiPL-E, MBPP, DS-1000, and APPS benchmarks.
• Advanced Code Completion Capabilities: A window size of 16K and a fill-in-the-blank task, supporting project-level code completion and infilling tasks.

The model supports a wide range of programming languages, making it a versatile tool for developers working in diverse environments.

DeepSeek V2.5: Combining General and Coding Capabilities

DeepSeek has also launched DeepSeek-V2.5, a powerful combination of DeepSeek-V2-0628 and DeepSeek-Coder-V2-0724. This new version retains the general conversational capabilities of the Chat model and the robust code processing power of the Coder model, while also better aligning with human preferences. Additionally, DeepSeek-V2.5 has seen significant improvements in tasks such as writing and instruction-following. It is available on both the web and API, with backward-compatible API endpoints.

Improvements in V2.5

DeepSeek-V2.5 outperforms both DeepSeek-V2-0628 and DeepSeek-Coder-V2-0724 on most benchmarks. In internal Chinese evaluations, DeepSeek-V2.5 shows a significant improvement in win rates against GPT-4o mini and ChatGPT-4o-latest, especially in tasks like content creation and Q&A. The model also shows improvements in the HumanEval Python and LiveCodeBench tests. Moreover, in the FIM completion task, the DS-FIM-Eval internal test set showed a 5.1% improvement, enhancing the plugin completion experience.

Impact and Accessibility

The release of new Deepseek version 3 features underscores the growing parity between open and closed-source AI models. It is fostering competition and reducing reliance on monopolistic players. The model is accessible under DeepSeek's license on GitHub, with an API available for enterprises. DeepSeek-V3’s impressive performance and affordability promise to accelerate innovation in AI development, offering enterprises versatile tools to enhance their AI-driven solutions.

Conclusion

DeepSeek V3 represents a significant advancement in the field of open-source AI. Its massive scale, efficient architecture, and cost-effective training make it a powerful and accessible tool for a wide range of applications. The model’s performance rivals that of closed-source models in many areas, and its specialized coding capabilities make it a valuable asset for developers. As DeepSeek continues to innovate, its models are poised to play an increasingly important role in the AI landscape.